<jupyter_start><jupyter_text><jupyter_code>import pandas as pd

df = pd.DataFrame({'animal': ['cat', 'cat', 'dog', 'cat', 'dog', 'cat', 'dog', 'dog'], 
                   'country': ['USA', 'Canada', 'Canada', 'Canada', 'USA', 'Canada', 'USA', 'USA'], 
                   'gender': ['male','female','male','male','male','male','female','female'],
                   'cat_integers': [1,1,0,1,0,1,0,0]})

df.head(8)
pd.crosstab(df['animal'], df['country'])
pd.crosstab(df['country'], df['gender'])
pd.crosstab(df['country'], df['gender'], normalize='columns')
pd.crosstab(df['country'], df['gender'], normalize='index')
pd.crosstab(df['country'], df['gender'], normalize=True )
pd.crosstab(df['country'],[df['gender'], df['animal']])
pd.crosstab(df['country'], [df['gender'], df['animal']], margins=True,).T

crosstab = pd.crosstab(df['country'], [df['gender'], df['animal']], margins=True,)

crosstab.hist();
crosstab.plot();<jupyter_output><empty_output><jupyter_text>Make featuresObjectives-  understand the purpose of feature engineering-  work with strings in pandas- work with dates and times in pandasLinks- [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)- Python Data Science Handbook  - [Chapter 3.10](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html), Vectorized String Operations  - [Chapter 3.11](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html), Working with Time Series- [Lambda Learning Method for DS - By <NAME>](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit?usp=sharing)<jupyter_code>!wget https://resources.lendingclub.com/LoanStats_2018Q4.csv.zip
!unzip LoanStats_2018Q4.csv.zip
!head LoanStats_2018Q4.csv
!tail LoanStats_2018Q4.csv<jupyter_output>"","","5600","5600","5600"," 36 months"," 13.56%","190.21","C","C1","","n/a","RENT","15600","Not Verified","Oct-2018","Current","n","","","credit_card","Credit card refinancing","836xx","ID","15.31","0","Aug-2012","0","","97","9","1","5996","34.5%","11","w","4404.61","4404.61","1701.34","1701.34","1195.39","505.95","0.0","0.0","0.0","Jul-2019","190.21","Aug-2019","Jul-2019","0","","1","Individual","","","","0","0","5996","0","0","0","1","20","0","","0","2","3017","35","17400","1","0","0","3","750","4689","45.5","0","0","20","73","13","13","0","13","","20","","0","3","5","4","4","1","9","10","5","9","0","0","0","0","100","25","1","0","17400","5996","8600","0","","","","","","","","","","","","N","","","","","","","","","","","","","","","N","","","","","",""
"","","23000","23000","23000"," 36 months"," 15.02%","797.53","C","C3","Tax Consultant","10+ years","MORTGAGE","75000","Source Verified","Oct-2018","Charged Off","n","","","debt_consolidation","Debt consolidation","352xx","AL","20.9[...]<jupyter_text>Load LendingClub datapandas documentation- [`read_csv`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)- [`options.display`](https://pandas.pydata.org/pandas-docs/stable/options.htmlavailable-options)<jupyter_code>import pandas as pd
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
df = pd.read_csv('LoanStats_2018Q4.csv', header=1, skipfooter=2, engine='python')
print(df.shape)
df.head()
df.tail()
df = df.drop(['id', 'member_id', 'url', 'desc'], axis=1) #dropping all the columns that have nothing but nan values (not there for privacy response)
df = df.drop(['hardship_loan_status','hardship_dpd' ,'hardship_reason' , 'hardship_status', 'deferral_term', 'hardship_amount', 'hardship_start_date','hardship_end_date' ,'hardship_length' ], axis=1)
df.isnull().sum().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Work with strings For machine learning, we usually want to replace strings with numbers.We can get info about which columns have a datatype of "object" (strings)<jupyter_code>df.dtypes
df.int_rate
#int_rate = '15.02%'
#int_rate[:-1]
#int_rate.strip('%')
type(int_rate.strip('%'))
example = float(int_rate.strip('%'))
example<jupyter_output><empty_output><jupyter_text>List Comprehension Example<jupyter_code>def remove_percent_to_float(string):
  return float(string.strip('%'))

  
int_list = ['15.02%', '17.08%', '13.80%']
[remove_percent_to_float(item) for item in int_list]
df['int_rate'] = df['int_rate'].apply(remove_percent_to_float)
df.head()
df.int_rate
df.int_rate.dtype
df['emp_title'].value_counts(dropna=False).head(20)
df['emp_title'].value_counts(dropna=False).head(10)
df['emp_title'].isnull().sum()
import numpy as np
type(np.NaN)
example = ['owner','Supervisor', 'Project Manager', np.NaN]

def clean_title(item): 
  if isinstance(item, str):
    return item.strip().title()
  else:
    return 'Unknown'

[clean_title(item) for item in example]
df['emp_title'] = df['emp_title'].apply(clean_title)
df.emp_title.value_counts(dropna=False).head(10)
df.emp_title.nunique()
pd.value_counts(df['emp_title'])
df.emp_title.nunique()<jupyter_output><empty_output><jupyter_text>Create `emp_title_manager`pandas documentation: [`str.contains`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html)<jupyter_code>df.emp_title_manager = True
print(df.emp_title_manager)
df['emp_title_manager'] = True
print(df['emp_title_manager'])
df['emp_title_manager'] = df['emp_title'].str.contains('Manager')
df.head()
#Dataframe Filtering Example
print(df.emp_title_manager)
condition = (df['emp_title_manager'] == True)
managers = df[condition]
print(managers.shape)
managers.head()
not_manager = df[df['emp_title_manager'] == False]
print(not_manager.shape)
not_manager.head()
managers['int_rate'].hist(bins=20)
not_manager['int_rate'].hist(bins=20);
managers.int_rate.plot.density();
not_manager.int_rate.plot.density();
managers.int_rate.mean()
not_manager.int_rate.mean()<jupyter_output><empty_output><jupyter_text>Work with dates<jupyter_code>df.issue_d.head()
df.issue_d.describe()
df.dtypes

df['issue_d'] = pd.to_datetime(df['issue_d'], infer_datetime_format=True)
df.issue_d.head()
df.issue_d.dt.year.head()
df.issue_d.dt.month
df['isssue_year'] = df.issue_d.dt.year
df['issue_month'] = df.issue_d.dt.month
df.head()
[col for col in df if col.endswith('_d')]
df.earliest_cr_line.head()
df.earliest_cr_line = pd.to_datetime(df['earliest_cr_line'], 
                                     infer_datetime_format=False)
df['days_from_earliest_credit_to_issue'] = (df['issue_d'] - df['earliest_cr_line']).dt.days
df.days_from_earliest_credit_to_issue.describe()
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.22998322403538204, 'fraction_numerical': 0.05307305170047278, 'mean_word_length': 8.070539419087137, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '386815', 'license': 'MIT', 'n_tokens_mistral': 2661, 'n_tokens_neox': 2409, 'n_words': 321, 'path': 'Pierre_Nelson_DS7_114_Make_Features.ipynb', 'repo_name': 'alxanderpierre/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '6560'}