<jupyter_start><jupyter_text>Engineering DatesDate variables are special type of categorical variable. By their own nature, date variables will contain a multitude of different labels, each one corresponding to a specific date and sometimes time. Date variables, when preprocessed properly can highly enrich a dataset. For example, from a date variable we can extract:- Month- Quarter- Semester- Day (number)- Day of the week- Is Weekend?- Hr- Time differences in years, months, days, hrs, etc.It is important to understand that date variables should not be used as the categorical variables we have been working so far when building a machine learning model. Not only because they have a multitude of categories, but also because when we actually use the model to score a new observation, this observation will most likely be in the future, an therefore its date label, will be different than the ones contained in the training set and therefore the ones used to train the machine learning algorithm.=============================================================================Let's look at how to pre-process date variables in a real life example. Lending Club**Lending Club** is a peer-to-peer Lending company based in the US. They match people looking to invest money with people looking to borrow money. When investors invest their money through Lending Club, this money is passed onto borrowers, and when borrowers pay their loans back, the capital plus the interest passes on back to the investors. It is a win for everybody as they can get typically lower loan rates and higher investor returns.If you want to learn more about Lending Club follow this link:https://www.lendingclub.com/The Lending Club dataset contains complete loan data for all loans issued through the 2007-2015, including the current loan status (Current, Late, Fully Paid, etc.) and latest payment information. Features (aka variables) include credit scores, number of finance inquiries, address including zip codes and state, and collections among others. Collections indicates whether the customer has missed one or more payments and the team is trying to recover their money.The file is a matrix of about 890 thousand observations and 75 variables. More detail on this dataset can be found in Kaggle's website: https://www.kaggle.com/wendykan/lending-club-loan-dataLet's go ahead and have a look!<jupyter_code>import pandas as pd
import numpy as np

import datetime
# let's load the Lending Club dataset with a few selected columns
# just a few rows to speed things up

use_cols = ['issue_d', 'last_pymnt_d']
data = pd.read_csv('loan.csv', usecols=use_cols, nrows=10000)
data.head()
# now let's parse the dates, currently coded as strings, into datetime format

data['issue_dt'] = pd.to_datetime(data.issue_d)
data['last_pymnt_dt'] = pd.to_datetime(data.last_pymnt_d)

data[['issue_d','issue_dt','last_pymnt_d', 'last_pymnt_dt']].head()
# Extracting Month from date

data['issue_dt_month'] = data['issue_dt'].dt.month

data[['issue_dt', 'issue_dt_month']].head()
data[['issue_dt', 'issue_dt_month']].tail()
# Extract quarter from date variable

data['issue_dt_quarter'] = data['issue_dt'].dt.quarter

data[['issue_dt', 'issue_dt_quarter']].head()
data[['issue_dt', 'issue_dt_quarter']].tail()
# We could also extract semester

data['issue_dt_semester'] = np.where(data.issue_dt_quarter.isin([1,2]),1,2)
data.head()
# day - numeric from 1-31

data['issue_dt_day'] = data['issue_dt'].dt.day

data[['issue_dt', 'issue_dt_day']].head()
# day of the week - from 0 to 6

data['issue_dt_dayofweek'] = data['issue_dt'].dt.dayofweek

data[['issue_dt', 'issue_dt_dayofweek']].head()
data[['issue_dt', 'issue_dt_dayofweek']].tail()
# day of the week - name

data['issue_dt_dayofweek'] = data['issue_dt'].dt.weekday_name

data[['issue_dt', 'issue_dt_dayofweek']].head()
data[['issue_dt', 'issue_dt_dayofweek']].tail()
# was the application done on the weekend?

data['issue_dt_is_weekend'] = np.where(data['issue_dt_dayofweek'].isin(['Sunday', 'Saturday']), 1,0)
data[['issue_dt', 'issue_dt_dayofweek','issue_dt_is_weekend']].head()
data[data.issue_dt_is_weekend==1][['issue_dt', 'issue_dt_dayofweek','issue_dt_is_weekend']].head()
# extract year 

data['issue_dt_year'] = data['issue_dt'].dt.year

data[['issue_dt', 'issue_dt_year']].head()
# perhaps more interestingly, extract the date difference between 2 dates

data['issue_dt'] - data['last_pymnt_dt']

# same as above capturing just the time difference
(data['last_pymnt_dt']-data['issue_dt']).dt.days.head()
# or the time difference to today, or any other day of reference

(datetime.datetime.today() - data['issue_dt']).head()
(datetime.datetime.today() - data['issue_dt']).tail()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12070056974045157, 'fraction_numerical': 0.0065414644439755225, 'mean_word_length': 5.980854197349043, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '472188', 'license': 'MIT', 'n_tokens_mistral': 1424, 'n_tokens_neox': 1361, 'n_words': 527, 'path': 'feature_engineering/12.1_Engineering_dates.ipynb', 'repo_name': 'Uttam580/basic_ml', 'size': '4739'}