<jupyter_start><jupyter_text>In-Class Project 1 Tuesday, April 26, 8:30-11 AM  Honesty PledgeThis project is open-Internet. I understand that I am allowed to use any existing resources on the web, including web search, but I am not allowed to consult with any people during this project. Giving and receiving help are both forbidden and will be prosecuted equally.Furthermore, I will not to discuss this in-class project with anyone until after 5 PM today, when everyone has completed the project. This includes not only students in the afternoon section, but also other students in this one.I understand that the penalty for violating this pledge is a grade of **F** in the course and disciplinary action by the Office of Student Rights & Responsibilities. Please sign below to indicate that you have read and understood this pledge. Your project will not be graded unless you sign below. <NAME>  DescriptionIn this in-class project, we will revisit the baby names data from lecture. The data can be found in the directory `/data/babynames/`. The files in this directory are all of the form `yobXXXX.csv`, where `XXXX` is a year. Each file consists of rows specifying a name, a gender, and a count of how many babies with that name and gender were born in the year `XXXX`.We will begin by looking at data from the year 2010.**Remember:** vectorization > for loop > manually writing out every case. Partial credit will be awarded even for brute-force solutions!<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
data = pd.read_csv("/data/babynames/yob2010.txt", names=["Name", "Gender", "Count"])

data<jupyter_output><empty_output><jupyter_text>Question 1 (10 points)How many babies born in 2010 (of either gender) had the same name as you? (Just write code in the cell below that outputs the final answer. No explanation required.)<jupyter_code># YOUR CODE HERE
#raise NotImplementedError()

data.ix[data['Name'] == 'Wilson','Count'].count()<jupyter_output><empty_output><jupyter_text>Question 2 (10 points)How many girls born in 2010 had names that began with the letter S?(Just write code in the cell below that outputs the final answer. No explanation required.)<jupyter_code># YOUR CODE HERE

data.ix[(data['Gender'] == 'F') & (data['Name'].str.startswith('S') == True), 'Count'].count()<jupyter_output><empty_output><jupyter_text>Question 3 (10 points)What was the most common first letter of names of babies born in 2010? What was the least common first letter? You may find `.str.get()` method of Pandas Series helpful.(Just write code in the cell below that outputs the final answer. No explanation required.)<jupyter_code># YOUR CODE HERE
data['first_letters'] = data['Name'].str.get(0).to_frame() #returns a column of beginning first letter

first_letter_counts = data.groupby('first_letters')[['Count']].count().sort_values(by = 'Count')

print(first_letter_counts.head())
print(first_letter_counts.tail())

del data['first_letters']<jupyter_output>Count
first_letters       
U                 69
X                109
Q                110
W                219
O                306
               Count
first_letters       
S               2335
M               2700
K               3117
J               3204
A               4410<jupyter_text>Question 4a (10 points)How many of the 2010 names can serve as both boys' and girls' names?(Just write code in the cell below that outputs the final answer. No explanation required.)<jupyter_code># YOUR CODE HERE
#data.ix[(data['Gender'] == 'F') & (data['Gender'] == 'M')].count() #check if both female and male, which is wrong


f_names = data.ix[(data['Gender'] == 'F') ]
m_names = data.ix[data['Gender'] == 'M']

gender_in_common = pd.merge(f_names,m_names, on = 'Name')
gender_in_common['Name'].count()<jupyter_output><empty_output><jupyter_text>Question 4b (10 points)Obviously, most mixed-gender names are very heavily skewed towards one gender or the other. For example, "William" is technically a mixed-gender name, but only 14 girls in 2010 were named William, compared with 16570 boys. Similarly, "Emma" is technically a mixed-gender name, but only 18 boys in 2010 were named Emma.If we want to know which are the most gender-ambiguous names, we should sort the names by the frequency of the less common gender. This way, even though William is a very common name overall, it would not be at the top of the list since the frequency of its less common gender (female) is only 14. Likewise, the frequency of the less common gender of Emma (male) is 18.Sort the mixed-gender names in decreasing order by the frequency of the less common gender. What are the Top 5 gender-ambiguous names?(Just write code in the cell below that outputs the final answer. No explanation required.)<jupyter_code># YOUR CODE HERE
# gender_in_common['less-common-freq'] = gender_in_common.ix[gender_in_common['Count_x'] > gender_in_common['Count_y'] \
#             | gender_in_common['Count_y'] > gender_in_common['Count_x'] ]  # IDEA

f_freq = list(gender_in_common['Count_x'])
m_freq = list(gender_in_common['Count_y'])
less_common_freq = []

arr_index = 0
last_index = gender_in_common.index[-1] # get last index

while(arr_index <= last_index):
    
    if f_freq[arr_index] < m_freq[arr_index]:
        less_common_freq.append(f_freq[arr_index])
    elif f_freq[arr_index] > m_freq[arr_index]:
        less_common_freq.append(m_freq[arr_index])
    else:
        less_common_freq.append(m_freq[arr_index]) #append either value will work; chose to append male count
        
    arr_index += 1


gender_in_common['less-common-freq']= less_common_freq
gender_in_common_sorted = gender_in_common.sort_values(by = 'less-common-freq', ascending = False )
gender_in_common_sorted.head(n = 5)<jupyter_output><empty_output><jupyter_text>Question 5a (10 points)The **entropy** of a distribution is a measure of how diverse its values are. A distribution which is usually one of a handful of values is said to have "low entropy", while a distribution which takes on many different values is said to have "high entropy".How diverse are human names? Let's calculate the entropy of the distribution of names in 2010. If a distribution takes on n values, each one with probability $p_i$, then the entropy $H$ is defined as $$ H = -\sum_{i=1}^n p_i \log_2(p_i). $$Calculate the entropy of the distribution of names in 2010.(Just write code in the cell below that outputs the final answer. No explanation required.)<jupyter_code># YOUR CODE HERE
from math import log

# def entropy2(labels):
#  """ Computes entropy of label distribution. """
#     n_labels = len(labels)

#     if n_labels <= 1:
#         return 0

#     counts = np.bincount(labels)
#     probs = counts / n_labels
#     n_classes = np.count_nonzero(probs)

#     if n_classes <= 1:
#         return 0

#     ent = 0.

#     # Compute standard entropy.
#     for i in probs:
#         ent -= i * log(i, base=n_classes)

#     return ent

# I am aware that probability != proportions, but will be using proportions to calculate entropy
# Wasn't sure how to calculate probability from n values

sample_size = data['Count'].sum() #overall total of people born
data['proportions'] = data['Count']/sample_size
data['entropy_calc'] = data['proportions'] * log(data['proportions'], 2)<jupyter_output><empty_output><jupyter_text>Question 5b (10 points)Calculate the entropy for every year between 1880 and 2014. Make a line plot of the entropy over time. Interpret what you see in context. YOUR ANSWER HERE  Question 6a (10 points)What percentage of boys born in 2000 were named "Michael"? What percentage of boys born in 2010 were named "Michael"? Is the difference statistically significant? Interpret your result in context.<jupyter_code>total_Michaels_2010 = data.ix[data['Name'] == 'Michael','Count'].count()
total_boys_2010 = data.ix[data['Gender'] == 'M','Count'].count()

Michaels_percentage_2010 = total_Michaels_2010 / total_boys_2010 * 100

print('Percentage of boys in 2010 that were named \'Michael\' : ' + str(Michaels_percentage_2010))
# for 2000:
data2000 = pd.read_csv("/data/babynames/yob2000.txt", names=["Name", "Gender", "Count"])

total_Michaels_2000 = data2000.ix[data2000['Name'] == 'Michael','Count'].count()
total_boys_2000 = data2000.ix[data2000['Gender'] == 'M','Count'].count()

Michaels_percentage_2000 = total_Michaels_2000 / total_boys_2000 * 100

print('Percentage of boys in 2000 that were named \'Michael\' : ' + str(Michaels_percentage_2000))
#=======================================================================================================

#ideas: 

#independent samples t-test
# confidence intervals?<jupyter_output>Percentage of boys in 2010 that were named 'Michael' : 0.0140439575872
Percentage of boys in 2000 that were named 'Michael' : 0.0165139129717<jupyter_text>Intuitively, the difference between the percentages is statistically insignificant because since the sample sizes are very large....    Question 6b (10 points)Make a bar graph showing the percentage of boys named Michael in 2000 and 2010. Include standard error bars.(Just write code in the cell below that outputs the final answer. No explanation required.)<jupyter_code># YOUR CODE HERE

#somehow need to get percentages to a dataframe
#with plt.style.context("fivethirtyeight"):
    #data.plot.bar(yerr=  )
    #height of bar(x-bar) = percentages of michaels
    #+- 1.96 * s/sqrt(n) ; right side is y-error<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10251552468161246, 'fraction_numerical': 0.030628354910009474, 'mean_word_length': 4.527632344386271, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '238289', 'license': 'MIT', 'n_tokens_mistral': 3011, 'n_tokens_neox': 2673, 'n_words': 1093, 'path': '.local/share/jupyter/nbgrader_cache/dlsun/wileong+Project-04-26+2016-04-26 18_00_26 UTC/In-Class Project 1 -- Morning Section.ipynb', 'repo_name': 'wileong/data_science_projects_directory', 'size': '9507'}