<jupyter_start><jupyter_text>_Lambda School Data Science_ Make featuresObjectives-  understand the purpose of feature engineering-  work with strings in pandas- work with dates and times in pandasLinks- [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)- Python Data Science Handbook  - [Chapter 3.10](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html), Vectorized String Operations  - [Chapter 3.11](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html), Working with Time Series  Get LendingClub data[Source](https://www.lendingclub.com/info/download-data.action)<jupyter_code>!wget https://resources.lendingclub.com/LoanStats_2018Q4.csv.zip
!unzip LoanStats_2018Q4.csv.zip
!head LoanStats_2018Q4.csv<jupyter_output>Notes offered by Prospectus (https://www.lendingclub.com/info/prospectus.action)
"id","member_id","loan_amnt","funded_amnt","funded_amnt_inv","term","int_rate","installment","grade","sub_grade","emp_title","emp_length","home_ownership","annual_inc","verification_status","issue_d","loan_status","pymnt_plan","url","desc","purpose","title","zip_code","addr_state","dti","delinq_2yrs","earliest_cr_line","inq_last_6mths","mths_since_last_delinq","mths_since_last_record","open_acc","pub_rec","revol_bal","revol_util","total_acc","initial_list_status","out_prncp","out_prncp_inv","total_pymnt","total_pymnt_inv","total_rec_prncp","total_rec_int","total_rec_late_fee","recoveries","collection_recovery_fee","last_pymnt_d","last_pymnt_amnt","next_pymnt_d","last_credit_pull_d","collections_12_mths_ex_med","mths_since_last_major_derog","policy_code","application_type","annual_inc_joint","dti_joint","verification_status_joint","acc_now_delinq","tot_coll_amt","tot_cur_bal","open_acc_6m","open_act_il","op[...]<jupyter_text>Load LendingClub datapandas documentation- [`read_csv`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)- [`options.display`](https://pandas.pydata.org/pandas-docs/stable/options.htmlavailable-options)<jupyter_code>import pandas as pd

#set pandas display options
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)

df = pd.read_csv('LoanStats_2018Q4.csv', header=1, skipfooter=2, engine='python')
print(df.shape)
df.head()
df.isnull().sum().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Work with strings For machine learning, we usually want to replace strings with numbers.We can get info about which columns have a datatype of "object" (strings)<jupyter_code>df = df.drop(columns=['id', 'member_id', 'desc', 'url'], axis='columns')
df.dtypes<jupyter_output><empty_output><jupyter_text>Convert `int_rate`Define a function to remove percent signs from strings and convert to floats<jupyter_code>type(df['int_rate'])
df['int_rate']<jupyter_output><empty_output><jupyter_text>Apply the function to the `int_rate` column<jupyter_code>#for loops are good but there are better ways
# for cell in df['int_rate']
# print(cell)
int_rate = '15.02%'
int_rate[:-1]
['15.02%', '13.56%', '16.91%']<jupyter_output><empty_output><jupyter_text>.apply() Function Example<jupyter_code>def remove_percent_to_float(string):
  return float(string.strip('%'))
int_list = ['15.02%', '13.56%', '16.91%']

[remove_percent_to_float(item) for item in int_list]
df['int_rate'] = df['int_rate'].apply(remove_percent_to_float)
df.head()<jupyter_output><empty_output><jupyter_text>Clean `emp_title`Look at top 20 titles<jupyter_code>df['emp_title'].value_counts(dropna=False).head(20)<jupyter_output><empty_output><jupyter_text>How many unique Employment Titles<jupyter_code>df['emp_title'].value_counts(dropna=False).reset_index().shape
df.describe(exclude='number')<jupyter_output><empty_output><jupyter_text>How often is `emp_title` null?<jupyter_code>df['emp_title'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Clean the title and handle missing values<jupyter_code>import numpy as np
type(np.NaN)
examples = ['owner', 'Supervisor', ' Project Manager', np.NaN]

def clean_title(item):
  if isinstance(item, str):
    return item.strip().title()
  else:
    return "Unknown"

[clean_title(item) for item in examples]
df['emp_title'] = df['emp_title'].apply(clean_title)

df.head()
df['emp_title'].value_counts(dropna=False).head(20)
df['emp_title'].describe(exclude='number')
df['emp_title'].nunique()<jupyter_output><empty_output><jupyter_text>Create `emp_title_manager`pandas documentation: [`str.contains`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html)<jupyter_code>df.emp_title_manager = True
print(df.emp_title_manager)<jupyter_output>True<jupyter_text>You have to use the square bracket column syntax when creating new dataframe columns<jupyter_code>df['emp_title_manager'] = True
print(df['emp_title_manager'])

df['emp_title_manager'] = df['emp_title'].str.contains("Manager")

df.head()<jupyter_output><empty_output><jupyter_text>Dataframe Filtering Example<jupyter_code>condition = (df['emp_title_manager'] == True)
managers = df[condition]
print(managers.shape)
managers.head()
managers = df[df['emp_title'].str.contains('Manager')]
print(managers.shape)
managers.head()
plebians = df[df['emp_title_manager'] == False]
print(plebians.shape)
plebians.head()
managers['int_rate'].hist(bins=20);
plebians['int_rate'].hist(bins=20);
managers['int_rate'].plot.density();
plebians['int_rate'].plot.density();
managers['int_rate'].mean()
plebians['int_rate'].mean()<jupyter_output><empty_output><jupyter_text>Work with dates pandas documentation- [to_datetime](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html)- [Time/Date Components](https://pandas.pydata.org/pandas-docs/stable/timeseries.htmltime-date-components) "You can access these properties via the `.dt` accessor"<jupyter_code>df['issue_d']
df['issue_d'].describe()
df['issue_d'].value_counts()
df.dtypes
df['issue_d'] = pd.to_datetime(df['issue_d'], infer_datetime_format=True)
df['issue_d'].head().values
df.dtypes
df['issue_d'].dt.year
df['issue_d'].dt.month
df['issue_year'] = df['issue_d'].dt.year
df['issue_month'] = df['issue_d'].dt.month
df.head()
[col for col in df if col.endswith('_d')]
df['earliest_cr_line'].head()
df['earliest_cr_line'] = pd.to_datetime(df['earliest_cr_line'],
                                        infer_datetime_format=True)
df['days_from_earliest_credit_to_issue'] = (df['issue_d'] - df['earliest_cr_line']).dt.days
df['days_from_earliest_credit_to_issue'].describe()<jupyter_output><empty_output><jupyter_text>ASSIGNMENT- Replicate the lesson code.- Convert the `term` column from string to integer.- Make a column named `loan_status_is_great`. It should contain the integer 1 if `loan_status` is "Current" or "Fully Paid." Else it should contain the integer 0.- Make `last_pymnt_d_month` and `last_pymnt_d_year` columns.  Converting the term column from str to int<jupyter_code>df['term'].head()
def remove_month(object):
  return int(object.strip('months'))
obj_list = ['24 months']

[remove_month(item) for item in obj_list]
df['term'] = df['term'].apply(remove_month)
df['term'].head()<jupyter_output><empty_output><jupyter_text>Making the loan_status_is_great column<jupyter_code>df['loan_status_is_great'] = [1 if x in 
                              ['Current', 'Fully Paid']
                              else 0 for x in
                              df['loan_status']]
df['loan_status_is_great'].head()<jupyter_output><empty_output><jupyter_text>Making last_pymnt_d_month and last_pymnt_d_year columns<jupyter_code>df['last_pymnt_d'] = pd.to_datetime(df['last_pymnt_d'], infer_datetime_format=True)
df['last_pymnt_d_month'] = df['last_pymnt_d'].dt.month
df['last_pymnt_d_month'].head()
df['last_pymnt_d_year'] = df['last_pymnt_d'].dt.year
df['last_pymnt_d_year'].head()<jupyter_output><empty_output><jupyter_text>STRETCH OPTIONSYou can do more with the LendingClub or Instacart datasets.LendingClub options:- There's one other column in the dataframe with percent signs. Remove them and convert to floats. You'll need to handle missing values.- Modify the `emp_title` column to replace titles with 'Other' if the title is not in the top 20. - Take initiatve and work on your own ideas!Instacart options:- Read [Instacart Market Basket Analysis, Winner's Interview: 2nd place, Kazuki Onodera](http://blog.kaggle.com/2017/09/21/instacart-market-basket-analysis-winners-interview-2nd-place-kazuki-onodera/), especially the **Feature Engineering** section. (Can you choose one feature from his bulleted lists, and try to engineer it with pandas code?)- Read and replicate parts of [Simple Exploration Notebook - Instacart](https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-instacart). (It's the Python Notebook with the most upvotes for this Kaggle competition.)- Take initiative and work on your own ideas! You can uncomment and run the cells below to re-download and extract the Instacart data<jupyter_code># !wget https://s3.amazonaws.com/instacart-datasets/instacart_online_grocery_shopping_2017_05_01.tar.gz
# !tar --gunzip --extract --verbose --file=instacart_online_grocery_shopping_2017_05_01.tar.gz
# %cd instacart_2017_05_01<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14463166990710738, 'fraction_numerical': 0.013933895009721323, 'mean_word_length': 7.82650142993327, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '281880', 'license': 'MIT', 'n_tokens_mistral': 3363, 'n_tokens_neox': 3208, 'n_words': 548, 'path': 'LS_DS_124_Make_features.ipynb', 'repo_name': 'moviedatascience/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '9258'}