<jupyter_start><jupyter_text>_Lambda School Data Science_ Make featuresObjectives-  understand the purpose of feature engineering-  work with strings in pandas- work with dates and times in pandasLinks- [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)- Python Data Science Handbook  - [Chapter 3.10](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html), Vectorized String Operations  - [Chapter 3.11](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html), Working with Time Series  Get LendingClub data[Source](https://www.lendingclub.com/info/download-data.action)<jupyter_code>!wget https://resources.lendingclub.com/LoanStats_2018Q4.csv.zip
!unzip LoanStats_2018Q4.csv.zip
!head LoanStats_2018Q4.csv<jupyter_output>Notes offered by Prospectus (https://www.lendingclub.com/info/prospectus.action)
"id","member_id","loan_amnt","funded_amnt","funded_amnt_inv","term","int_rate","installment","grade","sub_grade","emp_title","emp_length","home_ownership","annual_inc","verification_status","issue_d","loan_status","pymnt_plan","url","desc","purpose","title","zip_code","addr_state","dti","delinq_2yrs","earliest_cr_line","inq_last_6mths","mths_since_last_delinq","mths_since_last_record","open_acc","pub_rec","revol_bal","revol_util","total_acc","initial_list_status","out_prncp","out_prncp_inv","total_pymnt","total_pymnt_inv","total_rec_prncp","total_rec_int","total_rec_late_fee","recoveries","collection_recovery_fee","last_pymnt_d","last_pymnt_amnt","next_pymnt_d","last_credit_pull_d","collections_12_mths_ex_med","mths_since_last_major_derog","policy_code","application_type","annual_inc_joint","dti_joint","verification_status_joint","acc_now_delinq","tot_coll_amt","tot_cur_bal","open_acc_6m","open_act_il","op[...]<jupyter_text>Load LendingClub datapandas documentation- [`read_csv`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)- [`options.display`](https://pandas.pydata.org/pandas-docs/stable/options.htmlavailable-options)<jupyter_code>import pandas as pd
df = pd.read_csv('LoanStats_2018Q4.csv', header=1)
df.head(20)
pd.options.display.max_rows = 500
pd.options.display.max_columns = 500
df.isnull().sum().sort_values(ascending=False)
df.tail()
df = pd.read_csv('LoanStats_2018Q4.csv', skiprows=1, skipfooter=2, engine='python')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Work with strings For machine learning, we usually want to replace strings with numbers.We can get info about which columns have a datatype of "object" (strings)<jupyter_code>df.dtypes
df['int_rate']
df.describe()
df.describe(exclude='number')<jupyter_output><empty_output><jupyter_text>Convert `int_rate`Define a function to remove percent signs from strings and convert to floats<jupyter_code># '13.56%'.strip('%')  =>> '13.56' ; default is ' '; and is doing for the first or last caracter
# df['int_rate'] = df.int_rate.strip('%').float()

def remove_percent(string):
  return float(string.strip('%'))

df.int_rate = df.int_rate.apply(remove_percent)
df.int_rate.head()<jupyter_output><empty_output><jupyter_text>Apply the function to the `int_rate` column<jupyter_code><jupyter_output><empty_output><jupyter_text>Filtering the data frame<jupyter_code>df[df['term'] == ' 36 months'].shape
int_rate_greater_than_13 = df[df['int_rate'] >= 14.0]
int_rate_greater_than_13.reset_index()<jupyter_output><empty_output><jupyter_text>Clean `emp_title`Look at top 20 titles<jupyter_code>df.emp_title.value_counts().head(20)<jupyter_output><empty_output><jupyter_text>How often is `emp_title` null?<jupyter_code>df['emp_title'].isnull().sum()
df.emp_title.value_counts(dropna=False).head(20)
dtype(np.NaN)<jupyter_output><empty_output><jupyter_text>Clean the title and handle missing values<jupyter_code>import numpy as np

def clean_title(title):
  if isinstance(title, str):
    return title.strip().title()
  else:
    return 'Unknown'
  
  
df.emp_title = df.emp_title.apply(clean_title)
df.emp_title.head(20)<jupyter_output><empty_output><jupyter_text>Create `emp_title_manager`pandas documentation: [`str.contains`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html)<jupyter_code>df['emp_title_manager'] = df['emp_title'].str.contains('Manager')
managers = df[df['emp_title_manager'] == True]
plebians = df[df['emp_title_manager'] == False]
import seaborn as sns
sns.distplot(managers['int_rate'], bins=50);<jupyter_output><empty_output><jupyter_text>Work with dates pandas documentation- [to_datetime](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html)- [Time/Date Components](https://pandas.pydata.org/pandas-docs/stable/timeseries.htmltime-date-components) "You can access these properties via the `.dt` accessor"<jupyter_code>df['issue_d'].head().values
df['issue_d'].describe()
df['issue_d'] = pd.to_datetime(df['issue_d'], infer_datetime_format=True)
df['issue_d'].dt.year
df['issue_d'].dt.month
df['issue_year'] = df['issue_d'].dt.year
df.shape
df.head()
[col for col in df if col.endswith('_d')]
df['earlier_cr_line']<jupyter_output><empty_output><jupyter_text>ASSIGNMENT- Replicate the lesson code.- Convert the `term` column from string to integer.- Make a column named `loan_status_is_great`. It should contain the integer 1 if `loan_status` is "Current" or "Fully Paid." Else it should contain the integer 0.- Make `last_pymnt_d_month` and `last_pymnt_d_year` columns.  Convert the term column from string to integer.<jupyter_code>def term_to_int(string):
  for caracters in string.split():
    if caracters.isdigit() == True:
      return int(caracters)

print(type((term_to_int(' 36 months'))))

# [int(s) for s in df['term'].split() if s.isdigit()]
df['term'] = df['term'].apply(term_to_int)
df['term'].dtype<jupyter_output><empty_output><jupyter_text>Make a column named `loan_status_is_great`. It should contain the integer 1 if `loan_status` is "Current" or "Fully Paid." Else it should contain the integer 0.<jupyter_code>df['loan_status'].unique()
def loan_st(string):
  if (string.strip() == 'Current') or (string.strip() == 'Fully Paid'):
    return 1
  else:
    return 0
  

df['loan_status_is_great'] = df['loan_status'].apply(loan_st)
df['loan_status_is_great'].dtype
df[['loan_status','loan_status_is_great']]<jupyter_output><empty_output><jupyter_text>Make last_pymnt_d_month and last_pymnt_d_year columns.<jupyter_code>df['last_pymnt_d'].head()
df['last_pymnt_d'].isnull().sum()
df['last_pymnt_d'] = pd.to_datetime(df['last_pymnt_d'], infer_datetime_format=True)
df['last_pymnt_d'].head()
df['last_pymnt_d'].isnull().sum()
df['last_pymnt_d'].unique()
import numpy as np
import datetime as dt
import calendar
def new_column_month(date):
  if str(date) == 'NaT':
    return np.NaN
  else:
    return date.month
  
def new_column_year(date):
  if str(date) == 'NaT':
    return np.NaN
  else:
    return date.year
df['last_pymnt_d_month'] = df['last_pymnt_d'].apply(new_column_year)

df['last_pymnt_d_month'].head()
df['last_pymnt_d_month'] = df['last_pymnt_d_month'].astype(pd.Int64Dtype())
df['last_pymnt_d_month'].head()
df['last_pymnt_d_month'].isnull().sum()
df['last_pymnt_d_year'] = df['last_pymnt_d'].apply(new_column_year).astype(pd.Int64Dtype())

df['last_pymnt_d_year'].head()<jupyter_output><empty_output><jupyter_text>STRETCH OPTIONSYou can do more with the LendingClub or Instacart datasets.LendingClub options:- There's one other column in the dataframe with percent signs. Remove them and convert to floats. You'll need to handle missing values.- Modify the `emp_title` column to replace titles with 'Other' if the title is not in the top 20. - Take initiatve and work on your own ideas!Instacart options:- Read [Instacart Market Basket Analysis, Winner's Interview: 2nd place, Kazuki Onodera](http://blog.kaggle.com/2017/09/21/instacart-market-basket-analysis-winners-interview-2nd-place-kazuki-onodera/), especially the **Feature Engineering** section. (Can you choose one feature from his bulleted lists, and try to engineer it with pandas code?)- Read and replicate parts of [Simple Exploration Notebook - Instacart](https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-instacart). (It's the Python Notebook with the most upvotes for this Kaggle competition.)- Take initiative and work on your own ideas! You can uncomment and run the cells below to re-download and extract the Instacart data<jupyter_code># !wget https://s3.amazonaws.com/instacart-datasets/instacart_online_grocery_shopping_2017_05_01.tar.gz
# !tar --gunzip --extract --verbose --file=instacart_online_grocery_shopping_2017_05_01.tar.gz
# %cd instacart_2017_05_01<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14266039251692872, 'fraction_numerical': 0.01503500516469643, 'mean_word_length': 7.79313824419778, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 63, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '218490', 'license': 'MIT', 'n_tokens_mistral': 3237, 'n_tokens_neox': 3084, 'n_words': 552, 'path': 'Nicolae_Dubenco_DS4_124_Make_features.ipynb', 'repo_name': 'NikuDubenco/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '8713'}