<jupyter_start><jupyter_text>_Lambda School Data Science_ Make featuresObjectives-  understand the purpose of feature engineering-  work with strings in pandas- work with dates and times in pandasLinks- [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)- Python Data Science Handbook  - [Chapter 3.10](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html), Vectorized String Operations  - [Chapter 3.11](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html), Working with Time Series  Get LendingClub data[Source](https://www.lendingclub.com/info/download-data.action)<jupyter_code>!wget https://resources.lendingclub.com/LoanStats_2018Q4.csv.zip
!unzip LoanStats_2018Q4.csv.zip
# These are linux commands
!head LoanStats_2018Q4.csv
!tail LoanStats_2018Q4.csv<jupyter_output>"","","5600","5600","5600"," 36 months"," 13.56%","190.21","C","C1","","n/a","RENT","15600","Not Verified","Oct-2018","Current","n","","","credit_card","Credit card refinancing","836xx","ID","15.31","0","Aug-2012","0","","97","9","1","5996","34.5%","11","w","5083.61","5083.61","750.29","750.29","516.39","233.90","0.0","0.0","0.0","Feb-2019","190.21","Mar-2019","Feb-2019","0","","1","Individual","","","","0","0","5996","0","0","0","1","20","0","","0","2","3017","35","17400","1","0","0","3","750","4689","45.5","0","0","20","73","13","13","0","13","","20","","0","3","5","4","4","1","9","10","5","9","0","0","0","0","100","25","1","0","17400","5996","8600","0","","","","","","","","","","","","N","","","","","","","","","","","","","","","Cash","N","","","","","",""
"","","23000","23000","23000"," 36 months"," 15.02%","797.53","C","C3","Tax Consultant","10+ years","MORTGAGE","75000","Source Verified","Oct-2018","Charged Off","n","","","debt_consolidation","Debt consolidation","352xx","AL","[...]<jupyter_text>Load LendingClub datapandas documentation- [`read_csv`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)- [`options.display`](https://pandas.pydata.org/pandas-docs/stable/options.htmlavailable-options)<jupyter_code># Feautures are essentially another fancy term for specified columns
import pandas as pd
df = pd.read_csv('LoanStats_2018Q4.csv', skiprows=1, skipfooter=2, engine='python')
df.info() # That weird column is throwing things off
df.columns.tolist() # see all columns
df.isnull().sum().sum()
# How to extend rows
pd.options.display.max_columns = 500
pd.options.display.max_columns = 500
df.head()<jupyter_output><empty_output><jupyter_text>Work with strings For machine learning, we usually want to replace strings with numbers.We can get info about which columns have a datatype of "object" (strings)<jupyter_code>df.describe(exclude='number')<jupyter_output><empty_output><jupyter_text>Convert `int_rate`<jupyter_code>'13.56%'.strip('%') # How you strip symbols from a str
type('13.56%'.strip('%')) # still a string
float('13.56%'.strip('%')) # put all of this within a float and that accomplishes the job
# This is the one-liner that you really want to work with, most optimal

df['int_rate'] = df['int_rate'].str.strip('%').astype(float) 
#df['int_rate'].str.strip('%').astype(float).head() # Now it is a float, although not perm

 df['int_rate'].head() # Now it is a perm. float
#(df['int_rate'] / 100).head # Vectorizing the DF<jupyter_output><empty_output><jupyter_text>Define a function to remove percent signs from strings and convert to floats<jupyter_code>string = '13.56%'

def remove_percent(string):
  return float(string.strip('%'))

remove_percent(string)<jupyter_output><empty_output><jupyter_text>Apply the function to the `int_rate` column<jupyter_code># Both of those methods return the same results
# You got an error, because you already did it above

#df['int_rate'] = df['int_rate'].apply(remove_percent).head()<jupyter_output><empty_output><jupyter_text>Clean `emp_title`Look at top 20 titles<jupyter_code># Looking at the Top 20, just in the DS
df['emp_title'].head(20)

# Sometimes, you may want to combine titles 
df['emp_title'].value_counts().head(20) # These are the top 20 positions, organized<jupyter_output><empty_output><jupyter_text>How often is `emp_title` null?<jupyter_code>df['emp_title'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Clean the title and handle missing values<jupyter_code>import numpy as np
# I want to have consistant Capitalization
# Clean trailing values -- strip spaces
# Replace NaN with 'Missing'
# This can be used for various things, numbers etc

examples = ['owner', 'Supervisor ',
            ' Project Manager', np.nan]

def clean_title(x):
  if isinstance(x, str):
   
    return x.strip().title()
  
  else: 
    return 'Robot'
  
[clean_title(x) for x in examples]
# This is applying our previous function to all of the data in Emp Title
# Overwrite it & Assign back to that col
df['emp_title'] = df['emp_title'].apply(clean_title)
df['emp_title'].value_counts().head(20)<jupyter_output><empty_output><jupyter_text>Create `emp_title_manager`pandas documentation: [`str.contains`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html)<jupyter_code># Specify where Manager is
# Then add a new column to show where that is a true/ false col
df['emp_title_manager'] = df['emp_title'].str.contains('Manager')
# verify with shape
# Normalize shows percentage
df['emp_title_manager'].value_counts(normalize=True)
df.groupby('emp_title_manager')['int_rate'].mean()
df['emp_title'].value_counts().head()
# This shows how to represent that by a percentage
df.isnull().sum().sort_values(ascending=False) / len(df)<jupyter_output><empty_output><jupyter_text>Work with dates pandas documentation- [to_datetime](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html)- [Time/Date Components](https://pandas.pydata.org/pandas-docs/stable/timeseries.htmltime-date-components) "You can access these properties via the `.dt` accessor"<jupyter_code>df['issue_d'].head().values
df['issue_d'].describe()
df['issue_d'] = pd.to_datetime(df['issue_d'], infer_datetime_format=True)
df['issue_d'].head().values
df['issue_d'].describe()
# Specify year and month and create new cols
df['issue_year'] = df['issue_d'].dt.year
df['issue_month'] = df['issue_d'].dt.month
# Random Values
df['issue_month'].sample(n=10).values
# Time elapsed
#df['earliest_cr_line'].head()

df['earliest_cr_line'] = pd.to_datetime(df['earliest_cr_line'],
                                        infer_datetime_format=True)

# See how many days from ECL 
df['earliest_cr_line'].head()
# Difference between the 2 dates
# Really cool
df['days_from_earliest_credit_to_issue'] = (df['issue_d'] - df['earliest_cr_line']).dt.days

df['days_from_earliest_credit_to_issue'].describe()
25171 / 365
# Creating your own code to look for certain cols
[col for col in df if col.endswith('_d')]
for col in ['issue_d', 'last_pymnt_d', 'next_pymnt_d', 'last_credit_pull_d']:
  df[col] = pd.to_datetime(df[col], infer_datetime_format=True)
df.describe(include='datetime')<jupyter_output><empty_output><jupyter_text>ASSIGNMENT- Replicate the lesson code.- Convert the `term` column from string to integer. (*DONE)- Make a column named `loan_status_is_great`. It should contain the integer 1 if `loan_status` is "Current" or "Fully Paid." Else it should contain the integer 0. (*DONE)- Make `last_pymnt_d_month` and `last_pymnt_d_year` columns.(*DONE)  **Convert the 'term' column from string to integer**<jupyter_code># I did not use the .astype fucntion because the dtype was already an int, just needed months removed
df['term'] = df['term'].str.strip('months')
# It works!
df['term'].value_counts()<jupyter_output><empty_output><jupyter_text>**Make a column named 'loan_status_is_great'. It should contain:*   The integer 1 if 'loan_status' is 'Current' or 'Fully Paid'*   Else, it should contain 0<jupyter_code>df['loan_status'].value_counts(dropna=False)
# Kept getting an error when I tried to take this out of one-line 
# I am sure I could have simplified this by creating a dictionary or if/else 
# But this worked, so moving on

df['loan_status_is_great'] = df['loan_status'].replace('Current', 1).replace('Fully Paid', 1).replace('Late (31-120 days)', 0).replace('In Grace Period', 0).replace('Late (16-30 days)', 0 ).replace('Charged Off', 0)

# It worked!
df['loan_status_is_great'].value_counts()<jupyter_output><empty_output><jupyter_text>**Make 'last_pymnt_d_month' and 'last_pymnt_d_year' columns**<jupyter_code># Years Only
df['last_pymnt_d_year'] = df['last_pymnt_d'].dt.year

# Months Only
df['last_pymnt_d_month'] = df['last_pymnt_d'].dt.month
df['last_pymnt_d_year'].head()
df['last_pymnt_d_month'].head()<jupyter_output><empty_output><jupyter_text>STRETCH OPTIONSYou can do more with the LendingClub or Instacart datasets.LendingClub options:- There's one other column in the dataframe with percent signs. Remove them and convert to floats. You'll need to handle missing values.- Modify the `emp_title` column to replace titles with 'Other' if the title is not in the top 20. - Take initiatve and work on your own ideas!Instacart options:- Read [Instacart Market Basket Analysis, Winner's Interview: 2nd place, Kazuki Onodera](http://blog.kaggle.com/2017/09/21/instacart-market-basket-analysis-winners-interview-2nd-place-kazuki-onodera/), especially the **Feature Engineering** section. (Can you choose one feature from his bulleted lists, and try to engineer it with pandas code?)- Read and replicate parts of [Simple Exploration Notebook - Instacart](https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-instacart). (It's the Python Notebook with the most upvotes for this Kaggle competition.)- Take initiative and work on your own ideas! You can uncomment and run the cells below to re-download and extract the Instacart data<jupyter_code># !wget https://s3.amazonaws.com/instacart-datasets/instacart_online_grocery_shopping_2017_05_01.tar.gz
# !tar --gunzip --extract --verbose --file=instacart_online_grocery_shopping_2017_05_01.tar.gz
# %cd instacart_2017_05_01<jupyter_output><empty_output><jupyter_text>LendingClub<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Locate the only column left that contains % signs*     Remove the %*     Convert to floats*     Handle Missing Values<jupyter_code># I found which column by sheer luck, but I am sure there is another way to specify this
df['revol_util'].value_counts().head()
# Lets see how many Missing Values we are dealing with
df['revol_util'].isna().sum()
# I love this line of code
# In one shot, we removed the % and converted them to a float
df['revol_util'] = df['revol_util'].str.strip('%').astype(float) 
# Since we did not have data for this, I decided to replace this way
df['revol_util'] = df['revol_util'].replace(np.nan, '-')
# No more NaN
df['revol_util'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Modify the 'emp_title' column to replace titles with 'Other' if title is not in the top 20<jupyter_code>df['emp_title'].value_counts().head(20)
# Looking at only the top 20, and assigning a variable to it
top_counts = df['emp_title'].value_counts()[:21]

# Indexing
top_counts = top_counts.reset_index()

top_counts = top_counts['index'].as_matrix()
top_counts
# The coolest thing
top_counts = frozenset(top_counts)
top_counts
# Create a function for this
def the_others(x):
  if x in top_counts:
    return x
  else:
    return 'Other'
  
df['emp_title'] = df['emp_title'].apply(the_others)

# Final
df['emp_title'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1615795826866701, 'fraction_numerical': 0.03457492671150198, 'mean_word_length': 6.58600392413342, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 72, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '627734', 'license': 'MIT', 'n_tokens_mistral': 4206, 'n_tokens_neox': 3891, 'n_words': 947, 'path': 'module4-make-features/LS_DS_124_Make_features.ipynb', 'repo_name': 'tbradshaw91/DS-Unit-1-Sprint-2-Data-Wrangling', 'size': '11598'}