<jupyter_start><jupyter_text>5. Feature EngineeringSelect important features and construct more meaningful ones using the raw data that we have.In feature engineering, we try to generate additional features (variables) that have a better explanatory power or a higher predictive power with respect to our target variable.In our case, we first want to adjust our target variable.  Import Libraries<jupyter_code># Basic import(s)
import pandas as pd
import numpy as np

# Visualization libraries
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

# Feature selection
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_regression
from sklearn.model_selection import train_test_split

# Helper script(s)
from scripts.functions import get_line_equation, remove_outliers

# Hide all warnings in ipython
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Import Removed Outliers Datasets<jupyter_code># Loading of the known.csv dataset via pandas.
known_dataset = pd.read_csv('data/removed_outliers_with_IQR_known.csv')
# Loading of the unknown.csv dataset via pandas.
unknown_dataset = pd.read_csv('data/removed_outliers_with_IQR_unknown.csv')
# Show head for known_dataset.
# We want to make sure that the known.csv dataset has been read in correctly.
known_dataset.head()
# Show head for unknown_dataset.
# We want to make sure that the unknown.csv dataset has been read in correctly.
unknown_dataset.head()<jupyter_output><empty_output><jupyter_text>Preparation<jupyter_code># Import readed .csv datasets into DataFrames.
known_dataset=pd.DataFrame(known_dataset)
unknown_dataset=pd.DataFrame(unknown_dataset)
# Check whether the adjustments produce the desired result for known.csv dataset.
known_dataset.head()
# Check whether the adjustments produce the desired result for unknown.csv dataset.
unknown_dataset.head()<jupyter_output><empty_output><jupyter_text>Feature InteractionIt is often seen in ML experiments when two features combined through an arithmetic operation becomes more significant in explaining variances in the data, than the same two features separately. That's the reason why we analyze some feature variances below. Given question: "Why should you consider adding extra attributes like f2/f4 (f2 divided by f4) to the training data frame?"<jupyter_code># So we create a new variable that gives us the extra attribute f2/f4 (f2 divided f4) for known.csv dataset.
known_dataset['f2_div_f4'] = (known_dataset.f2/known_dataset.f4).round(2)
print(known_dataset['f2_div_f4'])
# So we create a new variable that gives us the extra attribute f2/f4 (f2 divided f4) for unknown.csv dataset.
unknown_dataset['f2_div_f4'] = (unknown_dataset.f2/unknown_dataset.f4).round(2)
print(unknown_dataset['f2_div_f4'])<jupyter_output>0       6.40
1      11.42
2       6.18
3       3.89
4       6.81
       ...  
762     5.44
763     4.83
764     5.84
765    20.55
766     5.77
Name: f2_div_f4, Length: 767, dtype: float64<jupyter_text>Given question: "Can we consider f6/f2 also as an additional extra attribute?"<jupyter_code># So we create a new variable that gives us the extra attribute for f6/f2 (f6 divided f2) for known.csv dataset.
known_dataset['f6_div_f2'] = (known_dataset.f6/known_dataset.f2).round(2)
print(known_dataset['f6_div_f2'])
# So we create a new variable that gives us the extra attribute for f6/f2 (f6 divided f2) for unknown.csv dataset.
unknown_dataset['f6_div_f2'] = (unknown_dataset.f6/unknown_dataset.f2).round(2)
print(unknown_dataset['f6_div_f2'])<jupyter_output>0      0.16
1      0.09
2      0.15
3      0.27
4      0.16
       ... 
762    0.19
763    0.23
764    0.20
765    0.20
766    0.18
Name: f6_div_f2, Length: 767, dtype: float64<jupyter_text>Given question: "How would you categorize the feature f5, in exactly 5 categories and add it to the data frame as f5_cat extra feature."<jupyter_code># To answer the question of how feature f5 can be categorised into exactly 5 features, let's first look again at the distributions with the function describe().
# We see that max is almost perfectly equal to 8.0409000, so we divide everything into 5 logical ticks: 1.6, 3.2, 4.8, 6.4 and 8.0409000.
known_dataset.f5.describe()

# So we create 5 subcategories (5 logical ticks: 1.6, 3.2, 4.8, 6.4 and 8.0409000) from the feature f5 (max value 8.0409000) for known.csv dataset.
f5_known_dataset = known_dataset['f5']

cond_list = [f5_known_dataset <= 1.6, f5_known_dataset <= 3.2, f5_known_dataset <= 4.8, f5_known_dataset <= 6.4, f5_known_dataset <= 8.0409000]
choice_list = [1, 2, 3, 4, 5] # New subcategories from 1-5.

known_dataset['f5_cat'] = np.select(cond_list, choice_list)
# Check if the new category f5_cat is created successfully for known dataset.
known_dataset.head(10)
known_dataset.info()
known_dataset['f5_cat']
# So we create 5 subcategories (5 logical ticks: 1.6, 3.2, 4.8, 6.4 and 8.0409000) from the feature f5 (max value 8.0409000) for unknown.csv dataset.
f5_unknown_dataset = unknown_dataset['f5']

cond_list = [f5_unknown_dataset <= 1.6, f5_unknown_dataset <= 3.2, f5_unknown_dataset <= 4.8, f5_unknown_dataset <= 6.4, f5_unknown_dataset <= 8.0409000]
choice_list = [1, 2, 3, 4, 5] # New subcategories from 1-5.

unknown_dataset["f5_cat"] = np.select(cond_list, choice_list)
# Check if the new category f5_cat is created successfully for unknown dataset.
unknown_dataset.head(10)
unknown_dataset.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 767 entries, 0 to 766
Data columns (total 13 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   longitude  767 non-null    float64
 1   latitude   767 non-null    float64
 2   f1         767 non-null    float64
 3   f2         767 non-null    float64
 4   f3         767 non-null    float64
 5   f4         767 non-null    float64
 6   f5         767 non-null    float64
 7   f6         767 non-null    float64
 8   t1         767 non-null    float64
 9   id         767 non-null    int64  
 10  f2_div_f4  767 non-null    float64
 11  f6_div_f2  767 non-null    float64
 12  f5_cat     767 non-null    int32  
dtypes: float64(11), int32(1), int64(1)
memory usage: 75.0 KB<jupyter_text>Small EDASmall EDA to answer the following questions for feature engineering.1. Given question: "Why should you consider adding extra attributes like f2/f4 (f2 divided by f4) to the training data frame?"2. Given question: "Can we consider f6/f2 also as an additional extra attribute?"3. Given question: "How would you categorize the feature f5, in exactly 5 categories and add it to the data frame as f5_cat extra feature."<jupyter_code>known_dataset.describe()
unknown_dataset.describe()
# Handle -inf values in f2_div_f4_cat and f6_div_f2_cat columns.
# Instead of dropping rows which contain any nulls and infinite numbers, 
# it is more succinct to the reverse the logic of that
# and instead return the rows where all cells are finite numbers.
# The numpy isfinite function does this and the '.all(1)' will only
# return a TRUE if all cells in row are finite.
known_dataset = known_dataset[np.isfinite(known_dataset).all(1)]
known_dataset.describe()
unknown_dataset.describe()<jupyter_output><empty_output><jupyter_text>Pie Chart<jupyter_code>fig = px.pie(known_dataset, values='t1', names='f5_cat')
fig.show()<jupyter_output><empty_output><jupyter_text>Pie Chart distribution for known.csvNew (sub-)categoriest1 datasamplespercentage (%) of known datasetf5_cat=180,173,516.762.79f5_cat=2790,230,675.727.5f5_cat=31,147,098,48939.9f5_cat=4607,232,532.621.1f5_cat=5253,448,066.18.81  Histograms<jupyter_code># Plotting of Histograms of known.csv dataset with new columns f2_div_f4, f6_div_f2 and f5_cat to compare with old columns.
known_dataset.hist(bins=50, figsize = (20,15)) 
plt.show()
%matplotlib inline
# Plotting of Histograms of unknown.csv dataset with new columns f2_div_f4, f6_div_f2 and f5_cat to compare with old columns.
unknown_dataset.hist(bins=50, figsize = (20,15)) 
plt.show()
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Detect And Remove Outliers - IQR Approach<jupyter_code># Analyse only those features.
columns_numeric =  ['f2_div_f4', 'f6_div_f2']
ax = sns.boxplot(data=known_dataset[columns_numeric])
ax = sns.boxplot(data=unknown_dataset[columns_numeric])
# Removing the outliers:
remove_outliers(known_dataset, columns_numeric)
remove_outliers(unknown_dataset, columns_numeric)
# Check again for outliers in known dataset:
ax = sns.boxplot(data=known_dataset[columns_numeric])
# Check again for outliers in unknown dataset:
ax = sns.boxplot(data=unknown_dataset[columns_numeric])
# Having replaced the outliers for known dataset with NaN, let us now check the sum of null values or missing values using the below code:
known_dataset.isnull().sum()
# Having replaced the outliers for unknown dataset with NaN, let us now check the sum of null values or missing values using the below code:
unknown_dataset.isnull().sum()<jupyter_output><empty_output><jupyter_text>Now, we can use any of the below techniques to treat the NULL values:<jupyter_code># So we decide again to impute missing values with Mean/Median, because columns in the dataset which are having numeric continous values can be replaced with the mean, median, or mode of remaining values in the column. This method can prevent the loss of data. Replacing two approximations (mean, median) is a statistical approach to handle the missing values.
known_dataset['f2_div_f4'] = known_dataset['f2_div_f4'].replace(np.NaN, known_dataset['f2_div_f4'].mean())
known_dataset['f6_div_f2'] = known_dataset['f6_div_f2'].replace(np.NaN, known_dataset['f6_div_f2'].mean())

unknown_dataset['f2_div_f4'] = unknown_dataset['f2_div_f4'].replace(np.NaN, unknown_dataset['f2_div_f4'].mean())
unknown_dataset['f6_div_f2'] = unknown_dataset['f6_div_f2'].replace(np.NaN, unknown_dataset['f6_div_f2'].mean())
# Now all is fine again for known dataset. No Null or NaN values exists.
known_dataset.isnull().sum()
# Now all is fine again for known dataset. No Null or NaN values exists.
unknown_dataset.isnull().sum()<jupyter_output><empty_output><jupyter_text>Heatmap<jupyter_code># Heatmap of the Pearson correlation coefficients of known.csv dataset.
mask = np.triu(known_dataset.corr())
plt.figure(figsize = (20,15))
ax = sns.heatmap(round(known_dataset.corr(), 2)
                 ,annot=True
                 ,mask=mask
                 ,cmap='RdBu_r')<jupyter_output><empty_output><jupyter_text>We can see some important relationships on the heatmap after we created the new features `f2_div_f4`, `f6_div_f2` and `f5_cat`:- f2_div_f4 & f6_div_f2 `very weak` relationship (score = `-0.71`),- f2_div_f4 & f5 `moderate` relationship (score = `0.5`),- f6_div_f2 & f5 `weak` relationship (score = `-0.54`),- f5_cat & f5 `extreme strong` relationship (score = `0.95`).- f5_cat & t1 `moderate` relationship (score = `0.56`),- f5_cat & f2_div_f4 `moderate` relationship (score = `0.32`),- f5_cat & f6_div_f2 `weak` relationship (score = `-0.54`).  Intercept And Slope<jupyter_code># Calculate the intercept and slope of a line between two points for known dataset.
known_dataset1 = [known_dataset.f1[1],known_dataset.t1[1]]
known_dataset2 = [known_dataset.f1[2],known_dataset.t1[2]]
slope, intercept = get_line_equation(p1=known_dataset1, p2=known_dataset2) 
print('intercept: ', intercept)
print('slope: ', slope)
# Plot line fitted on two points versus on all of the points of f2_div_f4 from known dataset.
fig = known_dataset.plot(x='f2_div_f4', 
                y='t1', 
                kind='scatter')
# Plot the linear regression line.
plt.plot(np.unique(known_dataset.f2_div_f4), np.poly1d(np.polyfit(known_dataset.f2_div_f4, known_dataset.t1, 1))(np.unique(known_dataset.f2_div_f4)))
# Plot the line between two points.
plt.plot([known_dataset1[0], known_dataset2[0]], [known_dataset1[1], known_dataset2[1]], color="red", linewidth=4)
plt.title("Line fitted on two points versus on all of the points")
# Plot line fitted on two points versus on all of the points of f6_div_f2 from known dataset.
fig = known_dataset.plot(x='f6_div_f2', 
                y='t1',
                kind='scatter')
# Plot the linear regression line.
plt.plot(np.unique(known_dataset.f6_div_f2), np.poly1d(np.polyfit(known_dataset.f6_div_f2, known_dataset.t1, 1))(np.unique(known_dataset.f6_div_f2)))
# Plot the line between two points.
plt.plot([known_dataset1[0], known_dataset2[0]], [known_dataset1[1], known_dataset2[1]], color="red", linewidth=4)
plt.title("Line fitted on two points versus on all of the points")
# Plot line fitted on two points versus on all of the points of f5_cat from known dataset.
fig = known_dataset.plot(x='f5_cat', 
                y='t1', 
                kind='scatter')
# Plot the linear regression line.
plt.plot(np.unique(known_dataset.f5_cat), np.poly1d(np.polyfit(known_dataset.f5_cat, known_dataset.t1, 1))(np.unique(known_dataset.f5_cat)))
# Plot the line between two points.
plt.plot([known_dataset1[0], known_dataset2[0]], [known_dataset1[1], known_dataset2[1]], color="red", linewidth=4)
plt.title("Line fitted on two points versus on all of the points")<jupyter_output><empty_output><jupyter_text>Feature Selection We are using Regression Feature Selection because we have numerical input and numerical output (label).Feature selection is performed using Pearson's Correlation Coefficient via the `f_regression()` function.<jupyter_code># The target variable t1 and the id are not necessary for our training dataset.
drop_lst = ['t1','id']
# We would like to consider all variables except the ones mentioned above.
all_features = [x for x in known_dataset.columns if x not in drop_lst]
# Check selected features for ML.
print(all_features)
# X contains all descriptive variables defined above.
X = known_dataset[all_features]
# We define y (our dependent variable).
y = known_dataset.t1
# Split into train and test sets.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)
# Summarize.
print('Train', X_train.shape, y_train.shape)
print('Test', X_test.shape, y_test.shape)<jupyter_output>Train (10496, 11) (10496,)
Test (4499, 11) (4499,)<jupyter_text>We define the SelectKBest class to use the `f_regression()` function and select, then transform the train and test sets.<jupyter_code># Configure to select all features.
fs = SelectKBest(score_func=f_regression, k='all')
# Learn relationship from training data.
fs.fit(X_train, y_train)
# Transform train input data.
X_train_fs = fs.transform(X_train)
# Transform test input data.
X_test_fs = fs.transform(X_test)
X_train_fs, X_test_fs, fs<jupyter_output><empty_output><jupyter_text>We can then print the scores for each variable (largest is better) and plot the scores for each variables as bar graph to get an idea of how many features we should select.<jupyter_code># What are scores for the features.
for i in range(len(fs.scores_)):
	print('Feature %d: %f' % (i, fs.scores_[i]))
# Plot the scores.
plt.figure(figsize=(20, 6))
plt.bar([i for i in range(len(fs.scores_))], fs.scores_)
plt.show()
# Check again selected features for ML to compare with feature scores above.
print(all_features)<jupyter_output>['longitude', 'latitude', 'f1', 'f2', 'f3', 'f4', 'f5', 'f6', 'f2_div_f4', 'f6_div_f2', 'f5_cat']<jupyter_text>We have obtained the score values for each variable in our training dataset and are now evaluating the results:- Feature 0: `not necessary` longitude (score = `1.064363`),- Feature 1: `not necessary` latitude (score = `0.000581`),- Feature 2: `not necessary` f1 (score = `32.220550`),- Feature 3: `important` f2 (score = `328.629710`),- Feature 4: `not necessary` f3 (score = `1.240831`),- Feature 5: `important` f4 (score = `111.432896`),- Feature 6: `very important` f5 (score = `5320.158971`),- Feature 7: `not necessary` f6 (score = `74.851346`),- Feature 8: `important` f2_div_f4 (score = `470.667044`),- Feature 9: `important` f6_div_f2 (score = `746.894718`),- Feature 10: `very important` f5_cat (score = `4687.550466`). We decide to use only those features for training that have a score of more than 100 points:- Feature 3: `important` f2 (score = `328.629710`),- Feature 5: `important` f4 (score = `111.432896`),- Feature 6: `very important` f5 (score = `5320.158971`),- Feature 8: `important` f2_div_f4 (score = `470.667044`),- Feature 9: `important` f6_div_f2 (score = `746.894718`),- Feature 10: `very important` f5_cat (score = `4687.550466`). In predictive modeling, we can therefore include the results of feature selection if we wish. However, this is not binding.  Export Created Features For Model Training<jupyter_code># Save featured known dataset with new features to a new .csv file
path = 'data/featured_known.csv'
known_dataset.to_csv(path,
index=False) # Avoid creating an index in a saved .csv file.
# Save featured unknown dataset with new features to a new .csv file
path = 'data/featured_unknown.csv'
unknown_dataset.to_csv(path,
index=False) # Avoid creating an index in a saved .csv file.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10201819878674755, 'fraction_numerical': 0.0550046663555763, 'mean_word_length': 4.969707520891365, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '208952', 'license': 'MIT', 'n_tokens_mistral': 6119, 'n_tokens_neox': 5618, 'n_words': 1843, 'path': '5_feature_engineering.ipynb', 'repo_name': 'dpedwards/ds-data-analysis-task', 'size': '17144'}