<jupyter_start><jupyter_text>My turn: What were the most popular male and female names in each year?<jupyter_code># get the data
baby_names = dw.load_dataset('nkrishnaswami/us-ssa-baby-names-national')
baby_names = baby_names.dataframes['names_ranks_counts']
# a silly observation
(baby_names.pivot_table(index='year',values='name',aggfunc='count') # count unique names per year
    .plot.line(title='Unique Names Per Year',legend=False) # plot it, uses panda
) 

(baby_names.sort_values(['year','sex','count'],ascending=False) # sort descending so most popular name first
     .groupby(['year','sex']) # group by year and gender 
     .agg(lambda x: x.iloc[0]) # keep the first (most popular) name each year
     ['name'] # keep only the name variable
     .unstack() # format wide
     [-20:]
)<jupyter_output><empty_output><jupyter_text>A side lesson: What on earth is **`lambda`**? Can you survive without learning `lambda`? Yes. But knowing `lambda` will make you a more powerful programmer, because it lets you define functions very quickly. And this is very useful! For example, the `agg` function can apply any function, not just built-in ones, and sometimes you'll want to use non built-in functions!So, generally, the syntax is ` = lambda  : `.<jupyter_code>my_fcn = lambda a : a*5 # if I call my_fcn(7), python will set a=7, then evalute the function a*5
print(my_fcn(7))<jupyter_output>35<jupyter_text>Now, in the example above inside `agg()` I never named the function. That's because it's unnecessary in that context, python knows to immediately call it.  Maybe you say "I hate that and will never ever write a lambda. How can I accomplish the problem above anyways?"Well, I'm sorry to hear that! Here is what you could do:<jupyter_code>def return_first_element(df):
    return df.iloc[0]

(baby_names.sort_values(['year','sex','count'],ascending=False) # sort descending so most popular name first
     .groupby(['year','sex']) # group by year and gender 
     .agg(return_first_element) # keep the first (most popular) name each year
     ['name'] # keep only the name variable
     .unstack() # format wide
     [-20:]
)<jupyter_output><empty_output><jupyter_text>My turn: Can we use the last letter of a name to predict the sex of the baby?<jupyter_code>(baby_names.query('year >= 1970') # let's focus on the last 50 years, maybe old-timey people had different naming rules
           .assign(last_letter = baby_names['name'].str[-1]) # get the last letter of each name string
           .assign(male = baby_names['sex'] == 'M') # = 1 if male, 0 else
           .assign(female= baby_names['sex'] == 'F') # = 1 if female, 0 else
           .pivot_table(index='last_letter',values=['male','female']) # average of male & female variables --> % of each
           .sort_values('male') # so the figure is in order
           .plot.barh(figsize=(7,7), # make it a bit bigger 
           title = 'Gender fraction by last letter in name \n \n "Feminine letters": A,H,E,I,Y')
           .set_ylabel("<-----Femine letters                         Masculine letters----->")
)<jupyter_output><empty_output><jupyter_text>Your turn: Practice Try each one alone, then check and compare with the person next to you.There are many ways to solve each, so if your approaches differ, explain it to each other!  Example 1<jupyter_code>start = datetime.datetime(2010, 1, 1) # you can specify start and end dates this way
end = datetime.datetime(2013, 1, 27)
macro_df = pdr.data.DataReader(['GDP','CPIAUCSL'], 'fred', start, end)<jupyter_output><empty_output><jupyter_text>- Q0: Do each of the four new golden rules for initial data exploration, from the lecture.- Q1: What is the second series above?- Q2: What is the frequency of the series?- Q3: What is the average ANNUAL GDP, based on the data?<jupyter_code># do your work here<jupyter_output><empty_output><jupyter_text>Example 2- Q4: Download the annual *real* gdp from 1960 to 2018 from FRED and compute the average annual percent change- Q5: Compute the average gdp percent change within *each decade*<jupyter_code># do your work here<jupyter_output><empty_output><jupyter_text>Example 3First, I'll load January data on unemployment, the Case-Shiller housing index, and median household income in three states (CA/MI/PA).<jupyter_code># LOAD DATA AND CONVERT TO ANNUAL

start = 1990 # pandas datareader can infer these are years
end = 2018
macro_data = pdr.data.DataReader(['CAUR','MIUR','PAUR', # unemployment 
                                  'LXXRSA','DEXRSA','WDXRSA', # case shiller index in LA, Detroit, DC (no PA  available!)
                                  'MEHOINUSCAA672N','MEHOINUSMIA672N','MEHOINUSPAA672N'], #  
                                 'fred', start, end)
macro_data = macro_data.resample('Y').first() # get's the first observation for each variable in a given year

# CLEAN UP THE FORMATING SOMEWHAT

macro_data.index = macro_data.index.year
print("\n\n DATA BEFORE FORMATTING: \n\n")
print(macro_data[:20]) # see how the data looks now? ugly variable names, but its an annual dataset at least
macro_data.columns=pd.MultiIndex.from_tuples([
    ('Unemployment','CA'),('Unemployment','MI'),('Unemployment','PA'),
    ('HouseIdx','CA'),('HouseIdx','MI'),('HouseIdx','PA'),
    ('MedIncome','CA'),('MedIncome','MI'),('MedIncome','PA')
    ])
print("\n\n DATA AFTER FORMATTING: \n\n")
print(macro_data[:20]) # this is a dataset that is "wide", and now 
                       # the column variable names have 2 levels - var name, 
                       # and unit/state that variable applies to<jupyter_output><empty_output><jupyter_text>- Q6: for each decade and state, report the average annual CHANGE (level, not percent) in unemployment- Q7: for each decade and state, report the average annual PERCENT CHANGE in house prices and household income<jupyter_code># do your work here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11563932755985736, 'fraction_numerical': 0.01358464934623875, 'mean_word_length': 4.0995670995671, 'pattern_counts': {'":': 1, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '563066', 'license': 'MIT', 'n_tokens_mistral': 1871, 'n_tokens_neox': 1753, 'n_words': 649, 'path': 'content/03/old 2.ipynb', 'repo_name': 'schemesmith/ledatascifi-2021', 'size': '5889'}