<jupyter_start><jupyter_text>Crosstab Review<jupyter_code>import pandas as pd

df = pd.DataFrame({'animal': ['cat', 'cat', 'dog', 'cat', 'dog', 'cat', 'dog', 'dog'], 
                   'country': ['USA', 'Canada', 'Canada', 'Canada', 'USA', "Canada",'USA', 'USA'], 
                   'gender': ['male','female','male','male','male','male','female','female'],
                   'cat_int': [1,1,0,1,0,1,0,0]})

df.head()
pd.crosstab(df['animal'],df['country'])#8
pd.crosstab(df['country'],df['gender'])
pd.crosstab(df['country'],df['gender'],normalize='columns') #Gives % of females and males - % by columns -
pd.crosstab(df['country'],df['gender'],normalize='index') #Gives % of countrys -% by rows-
pd.crosstab(df['country'],df['gender'],normalize=True)#More %s
pd.crosstab(df['country'],[df['gender'],df['animal']], margins=True)#different ways to put in 3, as well as tabulate totals
crosstab=pd.crosstab([df['country'],df['gender']],df['animal'], margins=True)
crosstab
crosstab.plot()<jupyter_output><empty_output><jupyter_text>Make featuresObjectives-  understand the purpose of feature engineering-  work with strings in pandas- work with dates and times in pandasLinks- [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)- Python Data Science Handbook  - [Chapter 3.10](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html), Vectorized String Operations  - [Chapter 3.11](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html), Working with Time Series- [Lambda Learning Method for DS - By <NAME>](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit?usp=sharing)  Get LendingClub data[Source](https://www.lendingclub.com/info/download-data.action)<jupyter_code>!wget https://resources.lendingclub.com/LoanStats_2018Q4.csv.zip
!unzip LoanStats_2018Q4.csv.zip
!head LoanStats_2018Q4.csv
!tail LoanStats_2018Q4.csv<jupyter_output>"","","5600","5600","5600"," 36 months"," 13.56%","190.21","C","C1","","n/a","RENT","15600","Not Verified","Oct-2018","Current","n","","","credit_card","Credit card refinancing","836xx","ID","15.31","0","Aug-2012","0","","97","9","1","5996","34.5%","11","w","4404.61","4404.61","1701.34","1701.34","1195.39","505.95","0.0","0.0","0.0","Jul-2019","190.21","Aug-2019","Jul-2019","0","","1","Individual","","","","0","0","5996","0","0","0","1","20","0","","0","2","3017","35","17400","1","0","0","3","750","4689","45.5","0","0","20","73","13","13","0","13","","20","","0","3","5","4","4","1","9","10","5","9","0","0","0","0","100","25","1","0","17400","5996","8600","0","","","","","","","","","","","","N","","","","","","","","","","","","","","","N","","","","","",""
"","","23000","23000","23000"," 36 months"," 15.02%","797.53","C","C3","Tax Consultant","10+ years","MORTGAGE","75000","Source Verified","Oct-2018","Charged Off","n","","","debt_consolidation","Debt consolidation","352xx","AL","20.9[...]<jupyter_text>Load LendingClub datapandas documentation- [`read_csv`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)- [`options.display`](https://pandas.pydata.org/pandas-docs/stable/options.htmlavailable-options)<jupyter_code>import pandas as pd 

#How to manipulate your view
pd.set_option('display.max_rows',500)
pd.set_option('display.max_columns',500)

df=pd.read_csv('LoanStats_2018Q4.csv', header=1,skipfooter=2, engine='python')
print(df.shape)
df.head()
df.tail()
df.isnull().sum().sort_values(ascending=False)#Some data is simply inapplicable vs. missing<jupyter_output><empty_output><jupyter_text>Work with strings For machine learning, we usually want to replace strings with numbers.We can get info about which columns have a datatype of "object" (strings)<jupyter_code>df = df.drop(columns=['id', 'member_id', 'desc', 'url'], axis='columns')#gets rid of superfluous columns
df.dtypes#remember to always re-write your df or save to new variable name to have both copies<jupyter_output><empty_output><jupyter_text>Convert `int_rate`Define a function to remove percent signs from strings and convert to floats<jupyter_code>type(df['int_rate'])#Its a series type
df['int_rate']<jupyter_output><empty_output><jupyter_text>Apply the function to the `int_rate` column<jupyter_code>int_rate = '15.02%'
int_rate[:-1]#gets rid of whats at the end of your string. Goes from beginning to one short of the end
int_list = ['15.02%', '13.56%', '16.91%']
int_list[:2]#Prints all things in the list past and including the second index
int_rate.strip('%')#another way to get rid of the % sign
type(int_rate.strip('%'))#but its still a string not an int
float(int_rate.strip('%'))#force change it
type(float(int_rate.strip('%')))#must still include float<jupyter_output><empty_output><jupyter_text>List comprehension example<jupyter_code>def remove_percent_to_float(string):#A function to remove all % and make string a float type
  return float(string.strip('%'))

int_list = ['15.02%', '13.56%', '16.91%']

[remove_percent_to_float(item) for item in int_list]<jupyter_output><empty_output><jupyter_text>.apply() function example<jupyter_code>df['int_rate'] = df['int_rate'].apply(remove_percent_to_float)#rewriting it into itself so that the dataframe doesnt have % in the set, it will remove the percent signs and change everything in the interest rate column
df.head()
#this is how we make changes to entire columns, using explicit functions<jupyter_output><empty_output><jupyter_text>Clean `emp_title`Look at top 20 titles<jupyter_code>df['emp_title'].value_counts(dropna=False).head(20)#chaining
df['emp_title'].value_counts(dropna=False).reset_index().shape<jupyter_output><empty_output><jupyter_text>How often is `emp_title` null?<jupyter_code>df['emp_title'].isnull().sum()
df.describe(exclude='number')
df['emp_title'].nunique()<jupyter_output><empty_output><jupyter_text>Clean the title and handle missing values<jupyter_code>import numpy as np
#3 different needs for clean up -at least- 

examples=['owner','Supervisor',' Project Manager', np.NAN]

def clean_title(item):
  if isinstance(item, str):
    return item.strip().title()
  else:
    return 'Unknown'
  

[clean_title(item) for item in examples]# mixed data type, needs all to be floats, also needs to deal with casess

df['emp_title']=df['emp_title'].apply(clean_title)
df.head()
df['emp_title'].nunique()#significantly different<jupyter_output><empty_output><jupyter_text>Create `emp_title_manager`pandas documentation: [`str.contains`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html)<jupyter_code>df['emp_title_manager']=True #sets it in list, not saving true to atribute 
print(df['emp_title_manager'])
#Clean your ROOM
title_list=['Manager', 'Sales Manager','Supervisor']


df['emp_title_manager'] = df['emp_title'].str.contains("Manager")#Overriding with Actual values

df.head()
#Really valuable to create new variables like this, can ask interesting questions
#dataframe filtering
condition = df['emp_title_manager']==True#All Manager positions 
managers=df[condition]
print(managers.shape)
managers.head()
plebians=df[df['emp_title_manager']==False]#All non-manager positions
print(plebians.shape)
plebians.head()
managers['int_rate'].hist(bins=20);
plebians['int_rate'].hist(bins=20)
managers['int_rate'].plot.density();
plebians['int_rate'].plot.density();
print(managers['int_rate'].mean())
print(plebians['int_rate'].mean())<jupyter_output>12.760582149647123
12.957292318828944<jupyter_text>Work with dates pandas documentation- [to_datetime](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html)- [Time/Date Components](https://pandas.pydata.org/pandas-docs/stable/timeseries.htmltime-date-components) "You can access these properties via the `.dt` accessor"<jupyter_code>df['issue_d']
df['issue_d'].describe()#only 3 unique fields, OCT, NOV, DEC
df['issue_d'].value_counts()#seasonality perhaps
df['issue_d']=pd.to_datetime(df['issue_d'], infer_datetime_format=True)
df['issue_d'].head().values#not look the same at all
df['issue_d'].dt.year #shows just year
df['issue_d'].dt.month
df['issue_year']=df['issue_d'].dt.year
df['issue_month']=df['issue_d'].dt.year
df.head()
[col for col in df if col.endswith('_d')]

df['earliest_cr_line'].head()
df['earliest_cr_line']=pd.to_datetime(df['earliest_cr_line'],infer_datetime_format=True)
(df['issue_d']-df['earliest_cr_line']).dt.days#reports out in days, can get rid of it and changes the type to int
df['days_from_earliest_credit_to_issue']=(df['issue_d']-df['earliest_cr_line']).dt.days
df['days_from_earliest_credit_to_issue'].describe()
25171/365#old<jupyter_output><empty_output><jupyter_text>Convert the term column from string to integer.Make a column named loan_status_is_great. It should contain the integer 1 if loan_status is "Current" or "Fully Paid." Else it should contain the integer 0.Make last_pymnt_d_month and last_pymnt_d_year columns. Convert Column from string to int<jupyter_code>#first figure out which is the term column
#I looked up all columns ending in term and found three but assume you just mean term 

df['term'].head()#first getting a feel for the column and whats in it
df['term'].describe()#they offer 2 types of loans in month type, perhaps we can use time
#df['term']=df['term'].dt.month
#df['term'].head() This ran an error that it was not datetimelike, so I will have to omit the months and convert whats left to ints. Was likely that I should not have made df['term''] equal itself, should have renamed it

def s2i (string):
  return int(string.strip(' months'))

x='36 months'
print(type(s2i(x)),s2i(x))#success


df['term']=[s2i(string) for string in df['term']]
df['term'].head()#Does what what intended

df['loan_status'].describe()#There are 7 unique answers. I want to make this into yes and no columns 
print(df['loan_status'].unique())
df['loan_status_is_great']=0#first we set the column to negative
print(df['loan_status_is_great'])
def loan_stat (string): 
  if string=='Current':
    return 1
  elif string=='Fully Paid':
    return 1
  else:
    return 0
 

df['loan_status_is_great'] = df['loan_status'].apply(loan_stat)

print(df['loan_status_is_great'])
  
  
df.head(18)
df['last_pymnt_d_month']=(pd.to_datetime(df['last_pymnt_d'],infer_datetime_format=True)).dt.month
#print(df['last_pymnt_d_month'])
df['last_pymnt_d_year']=(pd.to_datetime(df['last_pymnt_d'],infer_datetime_format=True)).dt.year
print(df['last_pymnt_d_year'])<jupyter_output>0         2019.0
1         2019.0
2         2019.0
3         2019.0
4         2019.0
5         2019.0
6         2019.0
7         2019.0
8         2019.0
9         2019.0
10        2019.0
11        2019.0
12        2019.0
13        2019.0
14        2019.0
15        2019.0
16        2019.0
17        2019.0
18        2019.0
19        2019.0
20        2019.0
21        2019.0
22        2019.0
23        2019.0
24        2019.0
25        2019.0
26        2019.0
27        2019.0
28        2019.0
29        2019.0
30        2019.0
31        2019.0
32        2019.0
33        2019.0
34        2019.0
35        2019.0
36        2019.0
37        2019.0
38        2019.0
39        2019.0
40        2019.0
41        2019.0
42        2019.0
43        2019.0
44        2019.0
45        2019.0
46        2019.0
47        2019.0
48        2019.0
49        2019.0
50        2019.0
51        2019.0
52        2019.0
53        2019.0
54        2019.0
55        2019.0
56        2019.0
57        2019.0
58        2019[...]
{'fraction_non_alphanumeric': 0.173701157427552, 'fraction_numerical': 0.07136019493516665, 'mean_word_length': 5.402228412256267, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '335528', 'license': 'MIT', 'n_tokens_mistral': 4534, 'n_tokens_neox': 3968, 'n_words': 860, 'path': 'My_work_up_of_LS_DS7_114_Make_Features.ipynb', 'repo_name': 'Madinanachan/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '11494'}