<jupyter_start><jupyter_text>2019-02-21 Exam General Instructions:Welcome to the **Python Programming (for Data Science)** exam session! Please, read **carefully** the instructions below before start writing code. This session will last **75 minutes** and is divided into **two parts**: one about "general" Python programming and the other about Python programming for Data Science. Each part is made of a set of exercises, which globally accounts for **16** + **16** = **32 points**.You will earn all of the points associated to an exercise **if and only if** the answer you provide passes successfully **all** the tests (both those that are visible and those that are hidden to you).To actually write down your implementation, make sure to fill in any place that says **_ YOUR CODE HERE_**. Note also that you should **either comment or delete** any **raise NotImplementedError()** exception.For this exam session **you will not be allowed** to use any lecture material yet you will be able to access the following APIs:-  [Python](https://docs.python.org/3.6/library/index.html)-  [Numpy](https://docs.scipy.org/doc/numpy-1.13.0/reference/)-  [Scipy](https://docs.scipy.org/doc/scipy-1.0.0/reference/)-  [Pandas](https://pandas.pydata.org/pandas-docs/version/0.22/api.html)-  [Matplotlib](https://matplotlib.org/2.1.1/api/index.html)-  [Seaborn](http://seaborn.pydata.org/api.html)Once you are done, save this notebook and rename it as follows:**YOURUSERNAME_2019-02-21.ipynb**where **YOURUSERNAME** is your actual username. To be consistent, we are expecting your username to be composed by your first name's initial, followed by your full lastname. As an example, in my case this notebook must be saved as **gtolomei_2019-02-21.ipynb** (Remember to insert an underscore **'_'** between your username and the date).Finally, go back to the [Moodle](https://elearning.studenti.math.unipd.it/esami/mod/assign/view.php?id=475) web page of the "**2019-02-21 Python Programming Exam**"; there, you will be able to upload your notebook file for grading.Submissions are allowed until Thursday, 21 February 2019 at 10:45 AMNote that there is no limit on the number of submissions; however, be careful when you upload a new version of this notebook because each submission overwrites the previous one. The due date indicated above is **strict**; after that, the system will not accept any more submissions and the latest uploaded notebook will be the one considered for grading.The archive you have downloaded (**2019-02-21-exam.tar**) is orgaized as follows:**2019-02-21-exam** (root)|----**2019-02-21.ipynb** (_this_ notebook)|----**corpus.txt** (the text corpus you will be using for answering general Python programming questions)|----**dataset.csv** (the dataset you will be using for answering data science related questions)|----**README.txt** (a description of the dataset above)... Now, sit back, relax, and do your best! **First Name** = Your _first name_ here**Last Name** = Your _last name_ here<jupyter_code>import math
import string
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# Adding the following line, allows Jupyter Notebook to visualize plots
# produced by matplotlib directly below the code cell which generated those.
%matplotlib inline
import seaborn as sns
from scipy.stats import pearsonr
from nose.tools import assert_equal
from operator import itemgetter

EPSILON = .0000001 # tiny tolerance for managing subtle differences resulting from floating point operations

TEXT_CORPUS_FILE = "corpus.txt"
DATASET_FILE = "dataset.csv"<jupyter_output><empty_output><jupyter_text>Part 1: General Coding (16 points)For **Part 1**, you will be asked to use the list below - called **corpus** - which contains a list of text documents, where each document is represented by a lowercase string with no punctuation character whatsoever. Please, execute the cell right below to successfully load those documents into **corpus**, see a few sample documents, and then answer the following questions.<jupyter_code># used to replace any punctuation symbol with an empty character ('')
translator = str.maketrans('', '', string.punctuation)
# load each individual document as a lowercase string into the list of strings `corpus`
corpus = [doc.strip().lower().translate(translator) for doc in open(TEXT_CORPUS_FILE)]
# print out the first 5 documents loaded
print("The following are the first 5 documents loaded out of a total of {} documents:\n".format(len(corpus)))
print("\n".join(corpus[:5]))<jupyter_output><empty_output><jupyter_text>Exercise 1.1 (1 point)Implement the function **avg_doc_length**, which returns the **average length** calculated over the documents in the **corpus**.We define the _length_ of a document the number of the tokens which the document string is made of; a _token_ is any substring which is separated from the other by a **whitespace character**, i.e., **" "**.(**EXAMPLE:** If the document you are working with is the string **"I think therefore I am"**, then the corresponding tokens will be: **"I"**, **"think"**, **"therefore"**, **"I"**, and **"am"** thereby the length of this document will be **5**.<jupyter_code>def avg_doc_length():
    """
    Returns the average length calculated over the documents in the `corpus`.
    """
    ### BEGIN SOLUTION
    return np.mean([len(doc.split(" ")) for doc in corpus])
    ### END SOLUTION
"""
Test the correctness of the implementation of the `avg_doc_length` function
"""

# Tests
assert_equal(True, avg_doc_length() < 15.6)
assert_equal(True, avg_doc_length() >= 4.2)
### BEGIN HIDDEN TESTS
assert_equal(True, np.abs(avg_doc_length() - 8.7028320702034296) < EPSILON)
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Exercise 1.2 (3 points)Implement the function **is_word_in_doc**, which takes as input a string **word** and and integer **doc_id**, and returns **True** if and only if the token **word** appears in the document **doc_id**. By convention, documents are identified by their index position in the **corpus** list, therefore the first element of the list will correspond to **doc_id = 0**, the second to **doc_id = 1**, and so on and so forth.The function must be _case-insensitive_, meaning that **is_word_in_doc("galileo", 42) = is_word_in_doc("Galileo", 42) = is_word_in_doc("GALILEO", 42)**. Finally, if the input **doc_id** is outside of its valid range $[0, N-1]$ (where $N$ is the total number of documents in the **corpus** list), the function should immediately return **False**.(**NOTE:** Words of documents contained in the **corpus** list are already lowercased, but there is no restiction on the **word** input to the **is_word_in_doc** function.)<jupyter_code>def is_word_in_doc(word, doc_id):
    """
    Return True iff the string `word` appears within the document `doc_id`, False otherwise.
    Plus, it returns False whenever `doc_id` is outside of its valid range of values [0, N-1] (N = len(corpus))
    """
    ### BEGIN SOLUTION
    if doc_id < 0 or doc_id >= len(corpus):
        return False
    return word.lower() in set([w for w in corpus[doc_id].split(" ")])
    ### END SOLUTION
"""
Test the correctness of the implementation of the `is_word_in_doc` function
"""

# Tests
assert_equal(True, is_word_in_doc("network", 4))
assert_equal(False, is_word_in_doc("Cable", 198))
assert_equal(False, is_word_in_doc("emails", 73))
assert_equal(True, is_word_in_doc("EMAIL", 73))
### BEGIN HIDDEN TESTS
assert_equal(True, is_word_in_doc("sciEncE", 0))
assert_equal(False, is_word_in_doc("galileo", -2))
assert_equal(False, is_word_in_doc("galileo", 2507))
assert_equal(False, is_word_in_doc("galileo", 2508))
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Exercise 1.3 (5 points)Implement the function **doc_stats**, which returns a custom data structure, i.e., dictionary, where each key is a **doc_id** and each value is a tuple containing the **min**, **max**, **mean**, and **standard deviation** (in this very specific order) of the number of characters of the words which _that_ **doc_id** is made of.For example, if the document is "I have been to Chargoggagoggmanchauggagoggchaubunagungamaugg lake last summer", then:-  **min = 1**-  **max = 45**-  **mean = 8.75**-  **std_dev = 13.77**(**NOTE:** The _Chargoggagoggmanchauggagoggchaubunagungamaugg_ lake truly exists, and is located in Webster, Massachussets, USA)<jupyter_code>def doc_stats():
    """
    Returns a dictionary where each key is a `doc_id` and each value is a tuple containing 
    the min, max, mean, and standard deviation of the number of characters of each word for that document.
    """
    doc_stats = {} # This is the variable that needs to be returned
    ### BEGIN SOLUTION
    for doc_id, doc in enumerate(corpus):
        doc_word_n_chars = [len(word) for word in doc.split(" ")]
        doc_stats[doc_id] = (np.min(doc_word_n_chars), 
                             np.max(doc_word_n_chars),
                             np.mean(doc_word_n_chars),
                             np.std(doc_word_n_chars)
                            )
    ### END SOLUTION
    return doc_stats
"""
Test the correctness of the implementation of the `doc_stats` function
"""

# Call off the function implemented above
stats = doc_stats()

# Tests
assert_equal(2, stats[0][0])
assert_equal(11, stats[0][1])
assert_equal(True, np.abs(7.25 - stats[0][2]) < EPSILON)
assert_equal(True, np.abs(3.2691742076555053 - stats[0][3]) < EPSILON)
### BEGIN HIDDEN TESTS
assert_equal(2507, len(stats))
assert_equal(1, stats[41][0])
assert_equal(15, stats[41][1])
assert_equal(True, np.abs(6.45454545455 - stats[41][2]) < EPSILON)
assert_equal(True, np.abs(3.79865134832 - stats[41][3]) < EPSILON)
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Exercise 1.4 (7 points)Implement the function **get_most_similar_docs**, which takes as input a document identifier **doc_i** and an integer n (**n > 0**), and returns an **ordered list of pairs**, where each pair is as follows: $(doc_j, sim_{i,j}), (j\neq i)$, i.e., the first element is a document identifier, whilst the second is the value of **Jaccard_similarity** computed between the set of $n$-grams of **doc_i** and **doc_j**. Such a list must be sorted by similarity (in not-ascending order) and by document identifier (in not-descending order).To compute Jaccard similarity between any two documents you firstly have to extract word $n$-grams out of those documents. For example, if the string document is "I really like python programming" then:-  **bi-grams** ($n$ = 2): **[("I", "really"), ("really", "like"), ("like", "python"), ("python", "programming")]**-  **tri-grams** ($n$ = 3): **[("I", "really", "like"), ("really", "like", "python"), ("like", "python", "programming")]**-  ...Finally, suppose $A$ and $B$ represents the sets of $n$-grams as extracted from **doc_i** and another document in the corpus **doc_j**, respectively. Then, the Jaccard similarity between $A$ and $B$ is computed as follows:$$J(A,B) = \frac{|A \cap B|}{|A \cup B|}$$**SUGGESTIONS:** Implement the following **two** helper functions: -  **n_grams(doc, n)** which takes as input the string representing a document and an integer n (**n > 0**), and extracts the $n$-grams from it, $n =$ **n** (pay attention to how the "sliding window" should move across the string in order to extract the corresponding substrings...)-  **jaccard_similarity(a, b)** which takes as input two sets a and b and returns the Jaccard similarity as specified above (to avoid 0 division error, the function returns 0 if **both** a and b are empty.)<jupyter_code>### SUGGESTION: Implement the function `n_grams` below which takes as input a string `doc` representing a document
### and an integer n > 0, and returns a list of tuples containing the n-grams of `doc`
def n_grams(doc, n):
    n_grams = []
    ### BEGIN SOLUTION
    words = doc.split(" ")
    n_grams = [tuple(words[i:i + n]) for i in range(len(words) - n + 1)]
    ### END SOLUTION
    return n_grams


### SUGGESTION: Implement the function `jaccard_similarity` below which takes as input two sets `a` and `b`
### and computes J = |a & b|/|a U b| (to avoid 0 division error, returns 0 if both a and b are empty)
def jaccard_similarity(a, b):
    if len(a) == 0 and len(b) == 0:
        return 0
    return len(set(a) & set(b))/len(set(a) | set(b))
    

def get_most_similar_docs(doc_i, n):
    """
    Returns a list of pairs (doc_j, n_grams_jaccard(doc_i, doc_j) [doc_i != doc_j] 
    sorted by n_grams_jaccard (from the highest to the lowest) and by doc_id.
    """
    most_similar_docs = [] # This is the variable that you shall return
    
    ### BEGIN SOLUTION
    d_i = corpus[doc_i]
    n_grams_doc_i = n_grams(d_i, n)
    
    for doc_j, doc in enumerate(corpus):
        if doc_j != doc_i:
            n_grams_doc_j = n_grams(doc, n)
            most_similar_docs.append((doc_j, jaccard_similarity(n_grams_doc_i, n_grams_doc_j)))
    
    most_similar_docs = sorted(most_similar_docs, key=itemgetter(0))
    most_similar_docs = sorted(most_similar_docs, key=itemgetter(1), reverse=True)
    ### END SOLUTION
    
    return most_similar_docs 
"""
Test the correctness of the implementation of the `get_most_similar_docs` function
"""

assert_equal(249, get_most_similar_docs(569, 2)[0][0])
assert_equal(True, np.abs(0.21428571428571427 - get_most_similar_docs(569, 2)[0][1]) < EPSILON)
assert_equal(504, get_most_similar_docs(27, 1)[1][0])
assert_equal(True, np.abs(0.2 - get_most_similar_docs(27, 1)[1][1]) < EPSILON)
### BEGIN HIDDEN TESTS
assert_equal(2506, len(get_most_similar_docs(111, 3)))
assert_equal(670, get_most_similar_docs(2047, 2)[0][0])
assert_equal(True, np.abs(0.18181818181818182 - get_most_similar_docs(2047, 2)[0][1]) < EPSILON)
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Part 2: Data Science (16 points)In this part, you will be working with the dataset file **dataset.csv**. For a complete description of this data source, please refer to the **README.txt** file included in the archive.In a nutshell, this dataset contains **721** unique Pokemons, including their number (ID), name, first and second type, and basic stats: HP, Attack, Defense, Special Attack, Special Defense, and Speed. Finally, it also shows whether the Pokemon is "legendary" or not.The cell below is responsible for correctly loading the dataset from the **dataset.csv** file. Once this is executed, you can start answering the questions below.<jupyter_code># Load the dataset stored at `DATASET_FILE` using "," as field separator and '?' to detect NAs
# and the specified columns as header

# column names used as header
colnames = ['id', 'name', 'type_1', 'type_2', 'total', 'hp', 'attack', 'defense', 
            'special_attack', 'special_defense', 'speed', 'generation', 'is_legendary']

# load dataset
data = pd.read_csv(DATASET_FILE, 
                   sep=',',
                   header=0,
                   names=colnames,
                   na_values='?')

# remove any duplicates
data = data.drop_duplicates('id', keep='first', inplace=False)
data.reset_index(inplace=True, drop=True)

print("Loaded `Pokemon` dataset into a dataframe of size ({} x {})".format(data.shape[0], data.shape[1]))

data.head()<jupyter_output><empty_output><jupyter_text>Exercise 2.1 (1 point)Implement the function **get_the_quickest** below. This takes as input a **pandas.DataFrame** object, and returns the record (i.e., the **pandas.Series**) of the quickest Pokemon in the collection (i.e., the one with the highest **speed**).<jupyter_code>def get_the_quickest(data):
    """
    Returns the record of the quickest Pokemon in the collection (i.e., the one with the highest speed)
    """
    ### BEGIN SOLUTION
    return data[data.speed == np.max(data.speed)]
    ### END SOLUTION
"""
Test the correctness of the implementation of the `get_the_quickest` function
"""

assert_equal("Ninjask", get_the_quickest(data)['name'].iloc[0])
assert_equal(45, get_the_quickest(data)['defense'].iloc[0])
assert_equal("Bug", get_the_quickest(data)['type_1'].iloc[0])
### BEGIN HIDDEN TESTS
assert_equal(False, get_the_quickest(data)['is_legendary'].iloc[0])
assert_equal(291, get_the_quickest(data)['id'].iloc[0])
assert_equal(456, get_the_quickest(data)['total'].iloc[0])
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Exercise 2.2 (3 points)Implement the function **attack_stats** below. This takes as input a **pandas.DataFrame** object and returns a tuple containing the min, max, avg, and median value of **attack** feature, yet computed on a _slice_ of the input **pandas.DataFrame**.The sliced dataset represents the subpopulation containing **legendary** Pokemons whose speed is **strictly above the overall average**, and whose defense ranges in $[52, 73)$.<jupyter_code>def attack_stats(data):
    """
    Returns a tuple containing the min, max, avg, and median value of `attack` feature,
    yet limited to a slice of the input DataFrame (data). 
    In particular, this slice will contain instances referring to legendary Pokemons
    whose speed is strictly above the overall average, and whose defense ranges in [52,73).
    """
    ### BEGIN SOLUTION
    sliced_data = data[(data.is_legendary == True) & 
                       (data.speed >= data.speed.mean()) & 
                       (data.defense >= 52) & 
                       (data.defense < 73)]

    return (sliced_data.attack.min(), 
            sliced_data.attack.max(), 
            sliced_data.attack.mean(), 
            sliced_data.attack.median())
    ### END SOLUTION
"""
Test the correctness of the implementation of the `attack_stats` function
"""

# Call off `attack_stats` function
stats = attack_stats(data)

assert_equal(90, stats[0])
assert_equal(125, stats[1])
### BEGIN HIDDEN TESTS
assert_equal(True, np.abs(111 - stats[2]) < EPSILON)
assert_equal(115.0, stats[3])
assert_equal(4, len(stats))
assert_equal(tuple, type(stats))
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Exercise 2.3 (5 points)Implement the function **get_top_k_strongest** below, which takes as input a **pandas.DataFrame** and an integer k, and returns the top-k Pokemon's **type_1** with the **highest average strength**. The overall strength is provided by the **total** column, as it contains a summary of all the statistics described in the other available columns.(**SUGGESTION:** In order to answer this question, you will need to compute the average strength for each group of Pokemon's **type_1**...)<jupyter_code>def get_top_k_strongest(data, k):
    ### BEGIN SOLUTION
    return data.groupby(by=['type_1'])['total'].mean().sort_values(ascending=False).index[:k]
    ### END SOLUTION
"""
Test the correctness of the implementation of the `get_top_k_strongest` function
"""

assert_equal(5, len(get_top_k_strongest(data, 5)))
assert_equal('Steel', get_top_k_strongest(data, 3)[1])
### BEGIN HIDDEN TESTS
assert_equal('Dark', get_top_k_strongest(data, 7)[-1])
assert_equal('Dragon', get_top_k_strongest(data, 1)[0])
assert_equal(0, len(get_top_k_strongest(data, 0)))
assert_equal('Poison', get_top_k_strongest(data, 20)[15])
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Exercise 2.4 (7 points)This exercise is made of **3** main questions, which you can answer independently to each other.  Question 1 (1 point)Feature **generation** represents an ordinal (numerical) variable which can take on 6 distinct values.Assign to the variable **lowest_generation** below the total number of first-generation Pokemons in the dataset.<jupyter_code>lowest_generation = None

### BEGIN SOLUTION
lowest_generation = data.generation.value_counts()[1]
### END SOLUTION
"""
Test the correctness of the `lowest_generation`
"""

assert_equal(False, (lowest_generation == None))
### BEGIN HIDDEN TESTS
assert_equal(151, lowest_generation)
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Question 2 (3 points)Plot the regression line of **special_attack** ($x$-axis, independent variable) against **special_defense** ($y$-axis, dependent variable) using **sns.regplot** and assign the result of the plot to the variable **reg_plot**. In addition to that, assign to the variable **pearson_r** the Pearson's correlation coefficient computed between the two random varibles (**special_attack** and **special_defense**). This can be computed by calling the **pearsonr** scipy's built-in function, which takes as input the two random variables and returns a **pair**: the first item is the Pearson's correlation coefficient, whilst the second item is the $p$-value associated to the computed statistic.<jupyter_code>reg_plot = None # assign this to the outcome of sns.regplot call
pearson_r = None # assign this to value of the Pearson's correlation coefficient

### BEGIN SOLUTION
# Create a Figure containing 1x1 subplots
fig, ax = plt.subplots(1, 1, figsize=(8,6))
# Regression plot line 'special_attack' vs. 'special_defense'
reg_plot = sns.regplot(x=data.special_attack, y=data.special_defense, color='#0099cc', ax=ax)
pearson_r = pearsonr(data.special_attack, data.special_defense)[0]
### END SOLUTION
"""
Test the correctness of `reg_plot` and `pearson_r`
"""

assert_equal(False, (reg_plot == None))
assert_equal(False, (pearson_r == None))
### BEGIN HIDDEN TESTS
assert_equal(True, np.abs(0.493037856827 - pearson_r) < EPSILON)
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>Question 3 (3 points)Plot the boxplot of how the variable **speed** is distributed across each of the 6 generations of Pokemons using **sns.boxplot** function. Then, assign to the list variable **outliers** (initially empty, i.e., **outliers = []**) the values of Pokemon generations which exhibit _any_ outlier. For example, if generations 1, 3, and 5 show some outliers then you should make the following assignment: **outliers = [1, 3, 5]** (order **doesn't** matter!).(**NOTE:** If there is no outlier in any Pokemon generation then you should leave the list **outliers** empty as it originally is.)<jupyter_code>box_plot = None # assign this to the outcome of sns.boxplot call
outliers = [] # change this according to the resulting box plot!

# Create a Figure containing 1x1 subplots
fig, ax = plt.subplots(1, 1, figsize=(8,6))
# Box plot 'speed' against 'generation'
### BEGIN SOLUTION
box_plot = sns.boxplot(x=data.generation, 
                       y=data.speed, 
                       palette=sns.color_palette("hls", n_colors=2), 
                       ax=ax)
outliers = [3, 6]
### END SOLUTION
"""
Test the correctness of `box_plot` and `outliers`
"""

assert_equal(False, (box_plot == None))
assert_equal(False, (outliers == None))
### BEGIN HIDDEN TESTS
assert_equal(set([3, 6]), set(outliers))
assert_equal(2, len(outliers))
### END HIDDEN TESTS<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12187650431053346, 'fraction_numerical': 0.021924642247604043, 'mean_word_length': 4.847492323439099, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 60, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '229227', 'license': 'MIT', 'n_tokens_mistral': 7507, 'n_tokens_neox': 6928, 'n_words': 2541, 'path': 'Exams/2018.2/2nd Exam/2019-02-21_Solution.ipynb', 'repo_name': 'GleydsonHudson/FIS-Exercises-Unipd', 'size': '22851'}