<jupyter_start><jupyter_text>Crosstab Review<jupyter_code>import pandas as pd

df = pd.DataFrame({'animal': ['cat', 'cat', 'dog', 'cat', 'dog', 'cat', 'dog', 'dog'], 
                   'country': ['USA', 'Canada', 'Canada', 'Canada', 'USA', 'Canada','USA', 'USA'], 
                   'gender': ['male','female','male','male','male','male','female','female']})
df.head()
pd.crosstab(df['animal'], df['country'])
# First argument becomes row index
pd.crosstab(df['animal'],df['gender'], normalize='columns')
pd.crosstab(df['animal'],df['country'], normalize='index')
# for more than two variables, add a list of multiple variables on the same dimension
crosstab = pd.crosstab(df['country'], [df['gender'],df['animal']])
pd.crosstab(df['country'], [df['gender'],df['animal']])
crosstab.plot()
# Crosstabs are dataframes like any other
pd.crosstab(df['country'], [df['gender'],df['animal']], margins=True)
pd.crosstab([df['country'],df['gender']],df['animal'])<jupyter_output><empty_output><jupyter_text>Make featuresObjectives-  understand the purpose of feature engineering-  work with strings in pandas- work with dates and times in pandasLinks- [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)- Python Data Science Handbook  - [Chapter 3.10](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html), Vectorized String Operations  - [Chapter 3.11](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html), Working with Time Series- [Lambda Learning Method for DS - By <NAME>](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit?usp=sharing)  Get LendingClub data[Source](https://www.lendingclub.com/info/download-data.action)<jupyter_code>!wget https://resources.lendingclub.com/LoanStats_2018Q4.csv.zip
!unzip LoanStats_2018Q4.csv.zip
!head LoanStats_2018Q4.csv<jupyter_output>Notes offered by Prospectus (https://www.lendingclub.com/info/prospectus.action)
"id","member_id","loan_amnt","funded_amnt","funded_amnt_inv","term","int_rate","installment","grade","sub_grade","emp_title","emp_length","home_ownership","annual_inc","verification_status","issue_d","loan_status","pymnt_plan","url","desc","purpose","title","zip_code","addr_state","dti","delinq_2yrs","earliest_cr_line","inq_last_6mths","mths_since_last_delinq","mths_since_last_record","open_acc","pub_rec","revol_bal","revol_util","total_acc","initial_list_status","out_prncp","out_prncp_inv","total_pymnt","total_pymnt_inv","total_rec_prncp","total_rec_int","total_rec_late_fee","recoveries","collection_recovery_fee","last_pymnt_d","last_pymnt_amnt","next_pymnt_d","last_credit_pull_d","collections_12_mths_ex_med","mths_since_last_major_derog","policy_code","application_type","annual_inc_joint","dti_joint","verification_status_joint","acc_now_delinq","tot_coll_amt","tot_cur_bal","open_acc_6m","open_act_il","op[...]<jupyter_text>Load LendingClub datapandas documentation- [`read_csv`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)- [`options.display`](https://pandas.pydata.org/pandas-docs/stable/options.htmlavailable-options)<jupyter_code>pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
df = pd.read_csv('LoanStats_2018Q4.csv', header=1, skipfooter=2, engine = 'python')
df.head()
# header = 1 because the first line is a note, skipfooter 2 to skip two notes
# at the bottom

df.shape
df.isnull().sum().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Work with strings For machine learning, we usually want to replace strings with numbers.We can get info about which columns have a datatype of "object" (strings)<jupyter_code>df.dtypes
# Learn data types of columns
df = df.drop(columns=['id', 'member_id', 'url', 'desc'])
print(df.shape)
df.isnull().sum().sort_values(ascending=False)<jupyter_output>(128412, 140)<jupyter_text>Convert `int_rate`Define a function to remove percent signs from strings and convert to floats<jupyter_code>df['int_rate']
int_rate = '15.02%'
# What can we do with this? Whatever we do, we can do with every cell, because series are iterable
int_rate[:-1]
# Just like a list comprehension
int_list = ['15.02%', '13.56%', '16.91']
int_list[:2]<jupyter_output><empty_output><jupyter_text>Apply the function to the `int_rate` column<jupyter_code>float(int_rate.strip('%'))
type(float(int_rate.strip('%')))
def remove_percent_to_float(string):
  return float(string.strip('%'))
[remove_percent_to_float(item) for item in int_list]
# .apply applies a function to an entire iterable
df['int_rate'] = df['int_rate'].apply(remove_percent_to_float)
df['int_rate']<jupyter_output><empty_output><jupyter_text>Clean `emp_title`Look at top 20 titles<jupyter_code>df['emp_title'].value_counts(dropna=False).head(20)<jupyter_output><empty_output><jupyter_text>How often is `emp_title` null?<jupyter_code>df['emp_title'].isnull().sum()
import numpy as np
type(np.NaN)<jupyter_output><empty_output><jupyter_text>Clean the title and handle missing values<jupyter_code>examples = ['owner', 'Supervisor', ' Project Manager', np.NaN]
def clean_title(item):
  if isinstance(item, str):
    return item.strip().title()
  else:
    return 'Unknown'
[clean_title(item) for item in examples]
df['emp_title'] = df ['emp_title'].apply(clean_title)
df.head()
df['emp_title'].value_counts(dropna=False).head(20)
df['emp_title'].value_counts(dropna=False).reset_index().shape
# Unique job titles<jupyter_output><empty_output><jupyter_text>Create `emp_title_manager`pandas documentation: [`str.contains`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html)<jupyter_code>df['emp_title_manager'] = True
title_list = ['Manager', 'Sales Manager', 'Supervisor']

for item in title_lists:
  print(item.str.contains('Manager'))
df['emp_title_manager'] = df['emp_title'].str.contains('Manager')
df.head()
# Dataframe Filtering
condition = df['emp_title_manager'] == True
managers = df[condition]
print(managers.shape)
managers.head()
# Make a new dataframe with only the managers
plebians = df[df['emp_title_manager'] == False]
print(plebians.shape)
plebians.head()
managers = df[df['emp_title'].str.contains('Manager')]
print(managers.shape)
managers.head()
# This code lets you filter without creating any columns
managers['int_rate'].hist()
plebians['int_rate'].hist()
managers['int_rate'].plot.density()
plebians['int_rate'].plot.density()
managers['int_rate'].mean()
plebians['int_rate'].mean()<jupyter_output><empty_output><jupyter_text>Work with dates pandas documentation- [to_datetime](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html)- [Time/Date Components](https://pandas.pydata.org/pandas-docs/stable/timeseries.htmltime-date-components) "You can access these properties via the `.dt` accessor"<jupyter_code>df['issue_d']
df['issue_d'].describe()
df['issue_d'].value_counts()
df.dtypes
df['issue_d'] = pd.to_datetime(df['issue_d'], infer_datetime_format=True)
df['issue_d'].head().values
df['issue_d'].dt.year
df['issue_d'].dt.month
df['issue_year'] = df['issue_d'].dt.year
df['isue_month'] = df['issue_d'].dt.month
df.head()
[col for col in df if col.endswith('_d')]
df['earliest_cr_line'] = pd.to_datetime(df['earliest_cr_line'], infer_datetime_format=True)
df['days_from_earliest_credit_to_issue'] = (df['issue_d'] - df['earliest_cr_line']).dt.days
df['days_from_earliest_credit_to_issue'].describe()
25171/365<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1684477691691157, 'fraction_numerical': 0.013224686080683943, 'mean_word_length': 8.786928104575164, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '484090', 'license': 'MIT', 'n_tokens_mistral': 2756, 'n_tokens_neox': 2608, 'n_words': 421, 'path': 'module4-makefeatures/LS_DS_114_Make_Features.ipynb', 'repo_name': 'cmgospod/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '7489'}