<jupyter_start><jupyter_text>14.1 1.USA.giv Data from Bitly<jupyter_code>path = 'datasets/bitly_usagov/example.txt'
open(path).readline()
import json

path = 'datasets/bitly_usagov/example.txt'
records = [json.loads(line) for line in open(path)]

records[0]<jupyter_output><empty_output><jupyter_text>Counting Time Zone in Pure PythonSuppose we were interested in finding the most commonly ocurring tie zone in the dataset.<jupyter_code>time_zones = [rec['tz'] for rec in records]
# Not all records have 'tz' field.
time_zones = [rec['tz'] for rec in records if 'tz' in rec]
time_zones[:10]<jupyter_output><empty_output><jupyter_text>Some of the time zone records are missing/unknown/empty.<jupyter_code># the python way; little difficult
def get_count(sequence):
    counts = {}
    for x in sequence:
        if x in count:
            count[x] += 1
        else:
            count[x] = 1
    return counts



# Using more advanced tools in Python standard library
# you can write the same thing more briefly

from collections import defaultdict

def get_counts(sequence):
    counts = defaultdict(int) # values will intiliaze to 0
    for x in sequence:
        counts[x] += 1
    return counts
counts = get_counts(time_zones)

print(counts['America/New_York'])
print(len(time_zones))
# If we wanted the top 10 time zones and their counts
def top_counts(count_dict, n=10):
    value_key_pairs = [(count, tz) for tz, count in count_dict.items()]
    value_key_pairs.sort()
    return value_key_pairs[-10:]

top_counts(counts)
# collections.Counter make this task a lot easier
from collections import Counter

counts = Counter(time_zones)
counts.most_common(10)<jupyter_output><empty_output><jupyter_text>Counting Time Zones with pandas<jupyter_code>import pandas as pd
import numpy as np

frame = pd.DataFrame(records)
frame.info()
frame.head()
frame.tz[:10]
tz_counts = frame['tz'].value_counts()
tz_counts[:10]<jupyter_output><empty_output><jupyter_text>Cleaning it up a little bit<jupyter_code>clean_tz = frame['tz'].fillna('Missing')
clean_tz[clean_tz == ''] = 'Unknown'
tz_counts = clean_tz.value_counts()
tz_counts[:10]<jupyter_output><empty_output><jupyter_text>Use Seaborn to visualize it<jupyter_code>import seaborn as sns
sns.set_theme()

subset = tz_counts[:10]
sns.barplot(y=subset.index, x=subset.values)<jupyter_output><empty_output><jupyter_text>The `a` field contains some information about the browser, device, or application used to perform URL shortening<jupyter_code>frame['a'].iloc[1]
frame['a'].iloc[50]
# A very long line
frame['a'].iloc[51][:50]<jupyter_output><empty_output><jupyter_text>Split off the the first token in the string (corrsonding roughly to the browser capability) and make another summary of the user behaviour.<jupyter_code>results = pd.Series([x.split()[0] for x in frame.a.dropna()])
results[:5]
results.value_counts()[:10]<jupyter_output><empty_output><jupyter_text>Suppose you want to decompose the top time zones into Windows and non-Windows users.<jupyter_code>cframe = frame[frame.a.notnull()] # exclude a's which are null
cframe = cframe.assign(os=np.where(cframe['a'].str.contains('Windows'), 'Windows', 'Not Windows'))
cframe.os[:5]<jupyter_output><empty_output><jupyter_text>Then group the data by its time zone column and this new list of operating systems:<jupyter_code>by_tz_os = cframe.groupby(['tz', 'os'])<jupyter_output><empty_output><jupyter_text>`size`The group counts, analogous to `value_counts`<jupyter_code>agg_counts = by_tz_os.size().unstack().fillna(0)
agg_counts[:10]<jupyter_output><empty_output><jupyter_text>Finally, let's select the top overall time zones. To do so, construct an indirect index array from the row counts in `agg_counts`.<jupyter_code># Use to sort in ascending order
indexer = agg_counts.sum(1).argsort()
indexer[5:]<jupyter_output><empty_output><jupyter_text>Use `take` to select the rows in that order, then slice off the last 10 rows(largest values).<jupyter_code>count_subset = agg_counts.take(indexer[-10:])
count_subset<jupyter_output><empty_output><jupyter_text>pandas has a convenience method called `nlargest` that does the same thing.<jupyter_code>agg_counts.sum(1).nlargest(10)<jupyter_output><empty_output><jupyter_text>Plot this in barplot.<jupyter_code># rearrange the data for plotting
count_subset = count_subset.stack()
count_subset.name  = 'total'
count_subset = count_subset.reset_index() # tz and os are index and integers are values.
count_subset
sns.barplot(x='total', y='tz', hue='os', data=count_subset)<jupyter_output><empty_output><jupyter_text>The plot doesn't make it easy to see the relative percentage of Windows users in the smaller groups, so let's normalize the group percentages to sum to 1.<jupyter_code>def norm_total(group):
    group['normed_total'] = group.total / group.total.sum()
    return group

results = count_subset.groupby('tz').apply(norm_total)

sns.barplot(x='normed_total', y='tz', hue='os', data=results)<jupyter_output><empty_output><jupyter_text>We could have computed the normalized sum more efficiently by using the `transform` method<jupyter_code>g = count_subset.groupby('tz')
results2 = count_subset.total / g.total.transform('sum')<jupyter_output><empty_output><jupyter_text>14.2 MovieLens 1M Dataset<jupyter_code>import pandas as pd

# Make display smaller 
pd.options.display.max_rows = 10

unames = ['user_id', 'gender', 'age', 'occupation', 'zip']
users = pd.read_table('datasets/movielens/users.dat', sep="::", header=None, names=unames)

rnames = ['user_id', 'movie_id', 'rating', 'timestamp']
ratings = pd.read_table('datasets/movielens/ratings.dat', sep="::", header=None, names=rnames)

mnames = ['movie_id', 'title', 'genres']
movies = pd.read_table('datasets/movielens/movies.dat', sep="::", header=None, names=mnames)

users.head()
ratings.head()
movies.head()
data = pd.merge(pd.merge(ratings, users), movies)
data
data.iloc[0]<jupyter_output><empty_output><jupyter_text>To get mean movie ratings for each film grouped by gender, we can use the `pivot_table` method.<jupyter_code>mean_ratings = data.pivot_table(values='rating', index='title', columns='gender', aggfunc='mean')
mean_ratings[:5]<jupyter_output><empty_output><jupyter_text>Now, let's try to filter down to movies that recieved at leat 250 ratings. For this group the data by title and use `size()` to get a Series of group sizes for each title.<jupyter_code>no_of_ratings_by_title =  data.groupby('title').size()
no_of_ratings_by_title[:10]
active_titles = no_of_ratings_by_title.index[ no_of_ratings_by_title >= 250 ]
active_titles<jupyter_output><empty_output><jupyter_text>The index of titles can then be used to select rowws from `mean_ratings`.<jupyter_code>mean_ratings = mean_ratings.loc[ active_titles ]
mean_ratings<jupyter_output><empty_output><jupyter_text>To see the top films among Female viewers, use `sort_values` to sort by the `F` column in descending order<jupyter_code>top_female_viewers = mean_ratings.sort_values(by='F', ascending=False)
top_female_viewers<jupyter_output><empty_output><jupyter_text>Measuring Rating DisagreementSuppose you wanted to find the movies that are most divisive between male and female viewers. One, way is to add a column to `mean_ratings` containing the difference in means, then sort by that:<jupyter_code>mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
sorted_by_diff = mean_ratings.sort_values(by='diff')

# Preferred by women (We substracted Female ratings from Male mean ratings)
sorted_by_diff<jupyter_output><empty_output><jupyter_text>Reversing the order gives the movies which the male prefers that the women didn'nt rate as highly<jupyter_code>sorted_by_diff[::-1][:10]<jupyter_output><empty_output><jupyter_text>Diagreement can be measured by the varaince or standard deviation of the ratings.<jupyter_code># Standard deviation of ratings grouped by title
ratings_std_by_title = data.groupby('title')['rating'].std()

# Filter down to active titles
ratings_std_by_title = ratings_std_by_title.loc[ active_titles ]

# Order series by values in descending order
ratings_std_by_title.sort_values(ascending=False)[:10]<jupyter_output><empty_output><jupyter_text>14.3 US Baby Names 1880-2010<jupyter_code>import pandas as pd

names1880 = pd.read_csv('datasets/babynames/yob1880.txt', names=['name', 'sex', 'births'])
names1880
# We can use the sum of the births column by sex
# as the total number of births in that year
names1880.groupby('sex')['births'].sum()<jupyter_output><empty_output><jupyter_text>`pandas.concat`Since the dataset is split into files by year, one of the first things to do is to assemble all of the data into a single DataFrame.<jupyter_code>years = range(1880, 2011)

pieces = []
columns = ['name', 'sex', 'births']

for year in years:
    path = 'datasets/babynames/yob{0}.txt'.format(year)
    frame = pd.read_csv(path, names=columns)
    frame['year'] = year
    
    pieces.append(frame)

# Concatenate everything into a single DataFrame
names = pd.concat(pieces, ignore_index=True)
names
total_births = names.pivot_table(values='births', index='year', columns='sex', aggfunc=sum)
total_births.tail()
total_births.plot(title='Total births by sex and year')<jupyter_output><empty_output><jupyter_text>Let's insert a column `prop` with the fraction of babies given each name relative to the toal number of births.A `prop` value of 0.02 would indicate that 2 out of every 100 babies were given a particular name.<jupyter_code>def add_prop(group):
    group['prop'] = group['births'] / group['births'].sum()
    return group

names = names.groupby(['year', 'sex']).apply(add_prop)

# prop for each year total births and particular sex
names<jupyter_output><empty_output><jupyter_text>When performing a group operations like this, it's often valuable to do a sanity check, like verifying that the `prop` column sums to 1 within all the groups.<jupyter_code>names.groupby(['year', 'sex'])['prop'].sum()<jupyter_output><empty_output><jupyter_text>Extract a subset of the data; For now, 1,000 names for each sex/year combination.<jupyter_code>def get_top1000(group):
    return group.sort_values(by='births', ascending=False)[:1000]

top1000 = names.groupby(['year', 'sex']).apply(get_top1000)
top1000.reset_index(inplace=True, drop=True)
top1000<jupyter_output><empty_output><jupyter_text>Analyzing Naming Trends<jupyter_code>boys = top1000[ top1000['sex'] == 'M' ]
girls = top1000[ top1000['sex'] == 'F']<jupyter_output><empty_output><jupyter_text>Let's form a pivot table of the total number of births by year and name:<jupyter_code>total_births = top1000.pivot_table(values='births', index='year', columns='name', aggfunc=sum)
total_births
total_births.info()
subset = total_births[['John', 'Harry', 'Mary', 'Marilyn']]
subset.plot(subplots=True, figsize=(12,10), grid=False, title='Number of births per year')<jupyter_output><empty_output><jupyter_text>Measuring the increase in naming diversityYou might believe that the these names have gone out of interest in the American population. But the story is a little different.One explanation for these decrease in plots is that fewer parents are choosing common names for their children. To confirm this hypothesis, let's measure the proportion of births represented by the top 1,000 most popular names.<jupyter_code># Here the aggregated sum is not zero 
# because we are looking at top 1000.
table = top1000.pivot_table(values='prop', index='year', columns='sex', aggfunc='sum')
table.plot(figsize=(12, 10),
           title='Sum of table1000.prop by year and sex', 
           yticks=np.linspace(0, 1.2, 13),
           xticks=range(1880, 2020, 10))<jupyter_output><empty_output><jupyter_text>Indeed, this is the case, the diversity is increasing. Another interesting metric is the number of distinct names, taken in order of popularity from highest to lowest, in the top 50% of births.<jupyter_code>df = boys[ boys.year == 2010 ]
df<jupyter_output><empty_output><jupyter_text>After sorting `prop` in descending order, we want to know how many of the most popular name it takes to reach 50%.<jupyter_code>prop_cumsum = df.sort_values(by='prop', ascending=False).prop.cumsum()
prop_cumsum[:10]
prop_cumsum.values.searchsorted(0.5) + 1<jupyter_output><empty_output><jupyter_text>Since arrays are zero indexed, this result gives you **117**. By contrast, in 1990 this number was much smaller.<jupyter_code>df = boys[ boys.year == 1900 ]
in1900 = df.sort_values(by='prop', ascending=False).prop.cumsum()
in1900.searchsorted(0.5) + 1<jupyter_output><empty_output><jupyter_text>You can now apply this operation to each year/sex combination, `groupby` those fields, and `apply` a function returning the count for each loop.<jupyter_code>def get_quantile_count(group, q=0.5):
    group = group.sort_values(by='prop', ascending=False).prop.cumsum()
    return group.searchsorted(q) + 1

diversity = top1000.groupby(['year', 'sex']).apply(get_quantile_count)
diversity = diversity.unstack('sex')
diversity.plot(title='Number of popular names in top 50%', figsize=(8, 7))<jupyter_output><empty_output><jupyter_text>Girl names have always been more diverse than boy names, and they have only become more so over time.  The "last letter" RevolutionIn 2007, baby name researcher <NAME> told that the distribution of boy names by final letter has changes significantely over the last 100 years.We first aggregate all of the births in the full dataset by year, sex, and final letter.<jupyter_code># extract last letter rom name column
get_last_letter = lambda x: x[-1]
names['last_letter'] = names['name'].map(get_last_letter)
#last_letters['name'] = 'last_letter'

table = names.pivot_table(values='births', index='last_letter', columns=['sex', 'year'], aggfunc='sum')
table.head()<jupyter_output><empty_output><jupyter_text>Then we select three representative years spanning the history.<jupyter_code>subtable = table.reindex(columns=[1910, 1960, 2010], level='year')
subtable.head()<jupyter_output><empty_output><jupyter_text>Next, normalize the table by total births to compute a new table containing proportion of total births for each sex ending in each letter.<jupyter_code>subtable.sum()
letter_prop = subtable / subtable.sum()
letter_prop<jupyter_output><empty_output><jupyter_text>We will now make barplot for each sex broken down by year.<jupyter_code>import matplotlib.pyplot as plt

fig, axes = plt.subplots(2, 1, figsize=(20, 8))
plt.subplots_adjust(hspace=0.3)

letter_prop['M'].plot(kind='bar', rot=0, ax=axes[0], title='Male')
letter_prop['F'].plot(kind='bar', rot=0, ax=axes[1], title='Female')<jupyter_output><empty_output><jupyter_text>Again normalize the full table by year and sex and select a subset of letters for boy names, finally transposing to make each column a time series:<jupyter_code>letter_prop = table / table.sum()
dny_ts = letter_prop.loc[['d', 'n', 'y'], 'M'].T
dny_ts.head()
dny_ts.plot(figsize=(8, 7))
ts_2 = letter_prop.T
unstacked_ts = ts_2.unstack(level=0)
unstacked_ts.plot(figsize=(20, 7))<jupyter_output><empty_output><jupyter_text>Boy names that became girl names (and vice versa)One example is the name Lesley or Leslie. Compute a list of names occuring in the datset starting with "lesl".<jupyter_code>all_names = pd.Series(top1000['name'].unique())
lesley_like = all_names[ all_names.str.lower().str.contains("lesl") ]
lesley_like<jupyter_output><empty_output><jupyter_text>From there, we can filter down to just those names and sum births grouped by name to see the relative frequencies:<jupyter_code>filtered = top1000[ top1000.name.isin(lesley_like) ]
filtered.groupby('name').births.sum()<jupyter_output><empty_output><jupyter_text>Next, let's aggregate by sex and year and normalize within year:<jupyter_code>table = filtered.pivot_table(values='births', index='year', columns='sex', aggfunc='sum')
table = table.div(table.sum(1), axis=0)
table.tail()
table.plot(style={'M': 'k', 'F':'k--'})<jupyter_output><empty_output><jupyter_text>14.4 USDA Food DatabaseThe US Department of Agriculture makes availiable a database of food nutrient information.<jupyter_code>import json

db = json.load(open('datasets/usda_food/database.json'))
len(db)<jupyter_output><empty_output><jupyter_text>- Each entry in `db` is a dict containing all the data for a single food. - The `'nutrient'` field is a list of dicts, one for each nutrien:<jupyter_code>db[0].keys()
db[0]['nutrients'][0]
nutrients = pd.DataFrame(db[0]['nutrients'])
nutrients[:7]<jupyter_output><empty_output><jupyter_text>When converting a list of dicts to a DataFrame, we can specify a list of fields to extract. We'll take the food names, groups, ID, and manufacturer:<jupyter_code>info_keys = ['description', 'group', 'id', 'manufacturer']
info = pd.DataFrame(db, columns=info_keys)
info.head()
info.info()
pd.value_counts(info['group'])[:5]<jupyter_output><empty_output><jupyter_text>To do some analysis, It's easiest to assemble the nutrients for each food into a single large table.Convert each list of food nutrients to a DataFrame, add a column for the food idmm and append the DataFrame to a list. Then, these can be concatenated together with `concat`.<jupyter_code># Some duplicates exist; drop them
nutrients.duplicated().sum() # no of duplicates
nutrients = nutrients.drop_duplicates()<jupyter_output><empty_output><jupyter_text>Since`'group'` and `'description'` are in both DataFrame object, we can rename for clarity:<jupyter_code>col_mapping = {
    'description': 'food',
    'group': 'fgroup'
}
info = info.rename(columns=col_mapping, copy=False)
info.info()
col_mapping = {
    'description': 'nutrient',
    'group': 'nutgroup'
}
nutrients = nutrients.rename(columns=col_mapping, copy=False)
nutrients<jupyter_output><empty_output><jupyter_text>We're ready to merge `info` with `nutrients`.<jupyter_code>ndata = pd.merge(nutrients, info, on='id', how='outer')
ndata.info()<jupyter_output><empty_output><jupyter_text>14.5 2012 Federal Election Commission Database<jupyter_code>fec = pd.read_csv('datasets/fec/P00000001-ALL.csv')
fec.info()
# a sample recod
fec.iloc[12345]<jupyter_output><empty_output><jupyter_text>You can see that there are no political party affliations in the data, so this would be useful to add.<jupyter_code>unique_cands = fec.cand_nm.unique()
unique_cands
unique_cands[2]<jupyter_output><empty_output><jupyter_text>One way to indicate party affiliation is using a dict:<jupyter_code>parties = {
    
}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11780331779247534, 'fraction_numerical': 0.01810690664642741, 'mean_word_length': 6.836448598130841, 'pattern_counts': {'":': 3, '<': 218, '<?xml version=': 0, '>': 219, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '54', 'dir': 'jupyter-structured-clean-dedup', 'id': '139794', 'license': 'Apache-2.0', 'n_tokens_mistral': 6495, 'n_tokens_neox': 6077, 'n_words': 1680, 'path': 'pydata/Ch 14 Data Analysis Examples.ipynb', 'repo_name': 'adhadse/ColabRepo', 'size': '18457'}