<jupyter_start><jupyter_text>Probability Ratio Encoding 1. Probability of Survived based on Cabin --- **Categorical Feature**2. Probability of Not Survived --- **1-prob(Survived)**3. **prob(Survived)/prob(Not Survived)**4. Dictonary to map cabin with probability.5. Replace with the categorical feature.<jupyter_code>import pandas as pd
df=pd.read_csv('titanic.csv',usecols=['Cabin','Survived'])
df.head()<jupyter_output><empty_output><jupyter_text>Replacing 'NAN' with 'Missing' Values<jupyter_code>df['Cabin'].fillna('Missing',inplace=True)
df.head()
df['Cabin'].unique()
df['Cabin']=df['Cabin'].astype(str).str[0]
df.head()
df.Cabin.unique()
prob_df=df.groupby(['Cabin'])['Survived'].mean()
prob_df = pd.DataFrame(prob_df)
prob_df
prob_df['Died']= 1 - prob_df['Survived']
prob_df.head()
prob_df['Probability_ratio']=prob_df['Survived']/prob_df['Died']
prob_df.head()
probability_encoded=prob_df['Probability_ratio'].to_dict()
probability_encoded
df['Cabin_encoded']=df['Cabin'].map(probability_encoded)
df.head(15)<jupyter_output><empty_output><jupyter_text>Transformation of the Features.1. Why Transformation of Features Are Required?    * Linear Regression --- Gradient Descent --- **Global Minima**    * Algorithms like KNN,K Means,Hierarichal Clustering --- **Eucledian Distance**2. Every Point has some vector and direction.3. Deep Learning Techniques(Standardization, Scaling --- 0-255 pixels)    * ANN ---> Global Minima, Gradient Descent    * CNN    * RNN  Types Of Transformation1. Normalization And Standardization2. Scaling to Minimum And Maximum values3. Scaling To Median And Quantiles4. Guassian Transformation    * Logarithmic Transformation    * Reciprocal Transformation    * Square Root Transformation    * Exponential Transformation    * Box-Cox Transformation.  Standardization* We try to bring all the variables or features to a similar scale. * Standarization means centering the variable at zero. * **Z = (x-x_mean)/std*** Mean = 0, Standard Deviation = 1.<jupyter_code>import pandas as pd
df=pd.read_csv('titanic.csv', usecols=['Pclass','Age','Fare','Survived'])
df.head()
df.isnull().sum()
df['Age'].fillna(df.Age.median(),inplace=True)
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Standarisation: We use the Standardscaler from sklearn Library.<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler=StandardScaler()


# fit vs fit_transform
df_scaled=scaler.fit_transform(df)
df_scaled
pd.DataFrame(df_scaled)
import matplotlib.pyplot as plt
%matplotlib inline
plt.hist(df_scaled[:,1],bins=20)
plt.xlabel('Pclass_Scaled')
plt.ylabel('Number of Points')
plt.hist(df_scaled[:,2],bins=20)
plt.xlabel('Age_Scaled')
plt.ylabel('Number of Points')
plt.hist(df_scaled[:,2],bins=20)
plt.xlabel('Fare_Scaled')
plt.ylabel('Number of Points')<jupyter_output><empty_output><jupyter_text>* **If there are outliers, it will affect the standardization**<jupyter_code>plt.hist(df['Fare'],bins=20)
plt.xlabel('Fare_Not_Scaled')
plt.ylabel('Number of Points')<jupyter_output><empty_output><jupyter_text>Min-Max Scaling (CNN) ---> Deep Learning Techniques* Min Max Scaling scales the values between 0 to 1. * X_scaled = (X - X.min / (X.max - X.min)<jupyter_code>from sklearn.preprocessing import MinMaxScaler

min_max=MinMaxScaler()
df_minmax=pd.DataFrame(min_max.fit_transform(df),columns=df.columns)

df_minmax.head()
plt.hist(df_minmax['Pclass'],bins=20)
plt.hist(df_minmax['Fare'],bins=20)
plt.hist(df_minmax['Age'],bins=20)<jupyter_output><empty_output><jupyter_text>Robust Scaler   1. It is used to scale the feature to median and quantiles.         2. Scaling using median and quantiles consists of substracting the median from all the observations, and then dividing by the interquantile difference.         3. The interquantile difference is the difference between the 75th and 25th quantile:      * **IQR = 75th quantile - 25th quantile**         4. X_scaled = (X - X.median) / IQR         5. 0,1,2,3,4,5,6,7,8,9,10      * **9** ---> 90 percentile ---> 90% of all values in this group is less than 9.      * **1** ---> 10 precentile ---> 10% of all values in this group is less than 1.<jupyter_code>from sklearn.preprocessing import RobustScaler

scaler=RobustScaler()
df_robust_scaler=pd.DataFrame(scaler.fit_transform(df),columns=df.columns)

df_robust_scaler.head()
plt.hist(df_robust_scaler['Fare'],bins=20)
plt.hist(df_robust_scaler['Age'],bins=20)
plt.hist(df_robust_scaler['Pclass'],bins=20)<jupyter_output><empty_output><jupyter_text>Guassian Transformation* Some machine learning algorithms like linear regression and logistic regression assume that the features are normally distributed ---> **Accuracy Performance increases when data is normally distributed**    * Logarithmic transformation    * Reciprocal transformation    * Square Root transformation    * Exponential transformation (more general, you can use any exponent)    * BoxCox transformation<jupyter_code>df=pd.read_csv('titanic.csv',usecols=['Age','Fare','Survived'])
df.head()<jupyter_output><empty_output><jupyter_text>Filling the missing "NAN" values with Median Values<jupyter_code>df['Age']=df['Age'].fillna(df['Age'].median())
df.isnull().sum()
import scipy.stats as stat

import pylab<jupyter_output><empty_output><jupyter_text>If you want to check whether feature is Guassian or Normal distributed we use --->  ***Q-Q plot***<jupyter_code>def plot_data(df, feature):
    plt.figure(figsize=(10, 6))
    
    plt.subplot(1, 2, 1)               # ----> 1 row, 2 column and 1st index.
    df[feature].hist()
    
    plt.subplot(1, 2, 2)               # ----> 1 row, 2 column and 2nd index.
    stat.probplot(df[feature], dist='norm', plot=pylab)
    
    plt.show()
plot_data(df, 'Age')<jupyter_output><empty_output><jupyter_text>* If all the points are falling in the red line, then we can say that the feature is normally distributed.<jupyter_code>plot_data(df, 'Fare')<jupyter_output><empty_output><jupyter_text>Logarithmic Transformation * **Logarithmic Transformation works best when your data is Right skewed or Left-skewed.**<jupyter_code>import numpy as np

df['Age_log'] = np.log(df['Age'])
plot_data(df, 'Age_log')<jupyter_output><empty_output><jupyter_text>Reciprocal Transformation<jupyter_code>df['Age_reciprocal']=1/df.Age

plot_data(df,'Age_reciprocal')<jupyter_output><empty_output><jupyter_text>Square Root Transformation<jupyter_code>df['Age_sqaure']=df.Age**(1/2)
plot_data(df,'Age_sqaure')<jupyter_output><empty_output><jupyter_text>Exponential Transformation<jupyter_code>df['Age_exponential']=df.Age**(1/1.2)
plot_data(df,'Age_exponential')<jupyter_output><empty_output><jupyter_text>Box-Cox Transformation * The Box-Cox transformation is defined as:    * **T(Y)=(Y exp(λ)−1)/λ*** where Y is the response variable and λ is the transformation parameter. λ varies from -5 to 5. In the transformation, all values of λ are considered and the optimal value for a given variable is selected.<jupyter_code>stat.boxcox(df['Age'])
df['Age_Boxcox'],parameters=stat.boxcox(df['Age'])
parameters
plot_data(df,'Age_Boxcox')<jupyter_output><empty_output><jupyter_text>'Fare' Variable Plots<jupyter_code>plot_data(df,'Fare')<jupyter_output><empty_output><jupyter_text>Logarithmic Transformation of (x+1)<jupyter_code>df['Fare_log']=np.log1p(df['Fare'])
plot_data(df,'Fare_log')
df['Fare_Boxcox'],parameters=stat.boxcox(df['Fare']+1)
plot_data(df,'Fare_Boxcox')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13701211305518168, 'fraction_numerical': 0.015074024226110363, 'mean_word_length': 6.521255060728745, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '511960', 'license': 'MIT', 'n_tokens_mistral': 2736, 'n_tokens_neox': 2552, 'n_words': 554, 'path': 'Feature Engineering/Feature_Engineering_Part_5.ipynb', 'repo_name': 'aryamanak10/Data-Science-Learning', 'size': '7430'}