<jupyter_start><jupyter_text>Analyzing Baby Names with Pandas In this exercise we analyze baby name popularity over the years using Python's popular pandas data science module.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
import pandas as pd<jupyter_output><empty_output><jupyter_text>Part 1: Parsing data After loading the required modules, let's take a look at the data files we're working with and load it into a data frame.<jupyter_code>with open('names/yob2016.txt','r') as file:
    yob2016 = file.readlines()
yob2016[:5]
names2016 = pd.read_csv('names/yob2016.txt', names=['name', 'sex', 'number'])
names2016.head()<jupyter_output><empty_output><jupyter_text>We make a list of data frames each corresponding to a year and concatenate them into one large data frame that contains all the data we'll be working with.<jupyter_code>names_all = []

for year in range(1880, 2017 + 1):
    names_all.append(pd.read_csv('names/yob{}.txt'.format(year), names=['name', 'sex', 'number']))
    names_all[-1]['year'] = year

allyears = pd.concat(names_all)
allyears.head()
allyears.tail()<jupyter_output><empty_output><jupyter_text>To look up names by popularity, it'll be convenient to index our data by sex, name, and year and sort it by this index.<jupyter_code>allyears_indexed = allyears.set_index(['sex', 'name', 'year']).sort_index()
allyears_indexed.head()<jupyter_output><empty_output><jupyter_text>Part 2: Some first plots of popularity Let's also create a function that plots a given name belonging to a particular sex.<jupyter_code>def plotname(sex, name):
    data = allyears_indexed.loc[sex, name]
    
    plt.plot(data.index, data.values)<jupyter_output><empty_output><jupyter_text>Here are some male names...<jupyter_code>plt.figure(figsize = (12, 3.5))

names = ['Maxim', 'Benjamin', 'Matthias', 'Erik']

for name in names:
    plotname('M', name)

plt.legend(names)
plt.title('Name popularity as a function of year')
plt.ylabel('Babies born')
plt.show()<jupyter_output><empty_output><jupyter_text>...and here some female ones.<jupyter_code>plt.figure(figsize = (12, 3.5))

names = ['Chiara', 'Claire', 'Clare', 'Clara', 'Ciara']

for name in names:
    plotname('F', name)

plt.legend(names)
plt.title('Name popularity as a function of year')
plt.ylabel('Babies born')
plt.show()<jupyter_output><empty_output><jupyter_text>If instead we would like to produce a stack plot, i.e., a plot where the values of the different functions are summed on top of each other, we need to index our data by year and turn the names into columns. The pandas method `.unstack` does just that.<jupyter_code>variants = allyears_indexed.loc['F'].loc[names].unstack(level = 0, fill_value = 0)
variants.head()<jupyter_output><empty_output><jupyter_text>`.unstack` sorts the new columns. To undo this behavior we call another method, `.reindex`.<jupyter_code>variants = variants.reindex(columns = names, level = 1)
variants.head()<jupyter_output><empty_output><jupyter_text>Now we're ready to pass the new data frame's index and values to pyplot. If you [look into the pyplot manual](https://matplotlib.org/2.1.1/api/_as_gen/matplotlib.pyplot.stackplot.html), you'll see that for a 1d array of dimension N passed as x-values, the 2d array passed for the y-values needs to be of dimension MxN. It follows that we need to transpose (`.T`) our data frame's values.<jupyter_code>plt.figure(figsize = (16, 5))

plt.stackplot(variants.index, variants.values.T, labels = names)
plt.legend(loc = 'upper left')
plt.title('Name popularity as a function of year (stack plot)')
plt.ylabel('Babies born')
plt.show()<jupyter_output><empty_output><jupyter_text>Part 3: Yearly Top-Tens The following function sorts a given year's data by popularity and keeps only the top ten names and ranking.<jupyter_code>def topten(sex, year):
    simple = allyears_indexed.loc[sex, :, year].sort_values('number', ascending = False).reset_index()
    simple = simple.drop(['sex', 'year', 'number'], axis = 1).head(10)
    
    simple.columns = [year]
    simple.index = simple.index + 1
    
    return simple
topten('M', 2009)<jupyter_output><empty_output><jupyter_text>Another function, and we can display as many years' most popular names side-by-side each other as we like.<jupyter_code>def toptens(sex, year0, year1):
    years = [topten(sex, year) for year in range(year0, year1 + 1)]
    
    return pd.concat(years, axis = 1)
    # could also use:
    # return years[0].join(years[1:])
toptens('M', 2000, 2010)
toptens('F',1985,1995)<jupyter_output><empty_output><jupyter_text>Let's see which name persisted in the list of top tens over a given period the longest and plot those names.<jupyter_code>toptens('F',1985,1995).stack().value_counts()
popular = toptens('F',1985,1995).stack().value_counts().index[:5]
plt.figure(figsize=(12,2.5))

for name in popular:
    plotname('F',name)
    
plt.legend(popular)
plt.title('The five most persistently popular names from 1985 to 1995 as seen from other decades')
plt.ylabel('Babies born')
plt.show()<jupyter_output><empty_output><jupyter_text>Indeed, out of these names, only "Elizabeth" is more than a fad.  Part 4: Finding fads To find fads more directly, we analyze the spikiness of each plot. In statistical terms, this can be measured by how much the sum of the squares of the annual name occurrences deviates from the square of their sum. The more localized the weight of a distribution is, the closer these two quantities will be to each other. For instance, if a name only occurs in a single year these two quantities are equal to each other. We begin by finding the total count a name occurred since record keeping began.<jupyter_code>totals = allyears.groupby(['sex', 'name'])['number'].sum()
totals.head()<jupyter_output><empty_output><jupyter_text>Similarly, we use a function to obtain the sum of the squares and divide these two quantities to obtain a "spikiness ratio."<jupyter_code>def sumsq(x):
    return sum(x ** 2)
spikiness = allyears.groupby(['sex', 'name'])['number'].agg(sumsq) / totals ** 2
spikiness.head()<jupyter_output><empty_output><jupyter_text>Let's only consider names that were at least somewhat common.<jupyter_code>spiky_common = spikiness[totals > 5000].copy()
spiky_common.sort_values(ascending = False, inplace = True)
spiky_common.head(10)
spiky_common.tail(5)
plt.figure(figsize=(12,3.5))

plotname('F','Louisa')
plotname('M','Shaquille')

plt.legend(['Louisa', 'Shaquille'])
plt.title('One of the most spiky ("faddy") against one of the least spiky names')
plt.ylabel('Babies born')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's plot the 10 most "faddy" names in history but exclude the current decade of record keeping since the spike of popularity for current names has not ended yet.<jupyter_code>totals_recent = allyears[allyears['year'] > 2009].groupby(['sex', 'name'])['number'].sum()
spiky_common = spikiness[(totals > 5000) & (totals_recent < 1000)].copy()
spiky_common.sort_values(ascending = False, inplace = True)
spiky_common.head(10)
fads = spiky_common.head(10).index.values<jupyter_output><empty_output><jupyter_text>We plot using a color jet to accommodate 10 different colors.<jupyter_code>plt.figure(figsize=(12,3.5))

ax = plt.subplot(111)

jet = plt.cm.jet
colors = jet(np.linspace(0, 1, 10))
ax.set_prop_cycle(color=colors)

for sex,name in fads:
    data = allyears_indexed.loc[sex, name]
    ax.plot(data.index, data.values)

plt.legend([name for sex,name in fads],loc='upper left')
plt.title('The top ten most spiky ("faddy") names in history')
plt.ylabel('Babies born')
plt.show()<jupyter_output><empty_output><jupyter_text>Part 5: The most popular unisex names To find the most popular unisex names, we begin by indexing our data by name and sex only and then unstack our data frame.<jupyter_code>names_by_sex = allyears.groupby(['name', 'sex'])['number'].sum().unstack(fill_value = 0)
names_by_sex.head()<jupyter_output><empty_output><jupyter_text>We then add columns for total counts across sexes and a ratio between them.<jupyter_code>names_ratio = names_by_sex.copy()
names_ratio['total'] = names_ratio.sum(axis = 1)
names_ratio['ratio'] = names_ratio['F'] / names_ratio['M']
names_ratio.head()<jupyter_output><empty_output><jupyter_text>This way we can limit our search to ratios of F:M or M:F of no more than 4.<jupyter_code>unisex_bool_mask = (names_ratio.loc[:, 'ratio'] > 0.25) & (names_ratio.loc[:, 'ratio'] < 4)<jupyter_output><empty_output><jupyter_text>We apply this boolean mask to our data fram and sort the result.<jupyter_code>unisex = names_ratio[unisex_bool_mask].copy()
unisex.sort_values('total', ascending = False, inplace = True)
unisex.head(10)<jupyter_output><empty_output><jupyter_text>We'll plot these names both, as they compare to each other's male/female version, and how popular they were when combining their male/female counts. To start with the latter, we need to slightly modify our initial data by summing equal names across genders.<jupyter_code>unisex_topten = unisex.head(10).index
allyears_nosex = allyears.groupby(['name', 'year'])['number'].sum()
allyears_nosex.head(10)<jupyter_output><empty_output><jupyter_text>We again plot using a color jet to accommodate 10 different colors.<jupyter_code>plt.figure(figsize = (12, 3.5))

ax = plt.subplot(111)

jet = plt.cm.jet
colors = jet(np.linspace(0, 1, 10))
ax.set_prop_cycle(color=colors)

for name in unisex_topten:
    data = allyears_nosex[name]
    ax.plot(data.index, data.values)

plt.legend([name for name in unisex_topten], loc = 'upper left')
plt.title('Combined totals of top ten unisex names with a ratio of at least 1:4')
plt.show()<jupyter_output><empty_output><jupyter_text>Lastly, we plot these names comparing their male/female popularity side-by-side.<jupyter_code>plt.figure(figsize = (12, 12))

for i,name in enumerate(unisex_topten):
    plt.subplot(5, 2, i + 1)
    
    plotname('M', name)
    plotname('F', name)
    
    plt.legend([name + ' M', name + ' F'],loc = 'upper left')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1235241591427721, 'fraction_numerical': 0.019744022224427026, 'mean_word_length': 5.857142857142857, 'pattern_counts': {'":': 0, '<': 103, '<?xml version=': 0, '>': 105, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '441569', 'license': 'CC0-1.0', 'n_tokens_mistral': 3611, 'n_tokens_neox': 3335, 'n_words': 1005, 'path': 'Baby Names.ipynb', 'repo_name': 'radheyshyamjangid/hcfest', 'size': '10079'}