<jupyter_start><jupyter_text>Project Name: Engineering Placements PredictionA University Announced Its On-Campus Placement Records For The Engineering Course. The Data Is From The Years 2013 And 2014.The Following Is The College Placements Data Compiled Over 2 years. Use This Data To Predict And Analyse Whether A Student Gets Placed, Based On His/Her Background.Perform Extensive EDAs And Bring Out Insights.The main aim of this project is to predict the whether student will be placed on their placement or not based on various features. This is a classification type problem. Here we will apply logistic regression, knn, decision tree, svm, random forest, xgboost etc.The purposes of this project is  to look into different features to observe their relationship, based on several features of individual such as age, stream, cgpa, gender if we can determine wherther they will be placed or not. Why this project may help someone?Let's say someone is in First or Second year and determining how much they should study for a good cgpa or not getting backlog or moreover to get placed. One aspect can be if any individual is havn't done any inernship then should they require internship or not.kaggle dataset [link](https://www.kaggle.com/tejashvi14/engineering-placements-prediction)  Details about dataset**Age**: ```Age At The Time Of Final Year```**Gender**: ```Gender Of Candidate, Male or Female```**Stream**```Engineering Stream That The Candidate Belongs To``````Computer Science, Information Technology, Mechanical,Electronics And Communication, Electrical, Civil```**CGPA**:```CGPA Till 6th Semester, Average of 6 sem's cgpa, min 0 and max 10```**Hostel**```Whether Student Lives In College Accomodation, 1->Yes and 0->No```**HistoryOfBacklogs**```Whether Student Ever Had Any Backlogs In Any Subjects, 1->Yes and 0->No```**PlacedOrNot**```Target Variable, 1->Yes and 0->No```  👨🏻‍💻 All the Lifecycle In A Data Science Projects1. 🔥 Exploratory Data Analysis(EDA))2. 📈 Feature Engineering3. 📝 Feature Selection4. 🤖 Model Building and Evaluation5. 🎊 Deployment  📥 Importing libraries<jupyter_code>import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns
from statsmodels.stats.outliers_influence import variance_inflation_factor
import scipy.stats as stat
import pylab 
from sklearn.model_selection import train_test_split
import xgboost as xgb
from sklearn.metrics import classification_report, confusion_matrix, plot_confusion_matrix, roc_auc_score
from sklearn.model_selection import GridSearchCV, KFold 
from sklearn.metrics import roc_auc_score
import joblib

%matplotlib inline
df = pd.read_csv("data/collegePlace.csv")
df.head()<jupyter_output><empty_output><jupyter_text>📝Metadeta about dataset<jupyter_code>print(f"dataset contains {df.shape[0]} rows and {df.shape[1]}")
df.describe()<jupyter_output><empty_output><jupyter_text>Datatypes of each columns<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>🔥 Exploratory Data Analysis(EDA)¶In Exploratory Data Analysis We will Analyze To Find out the below stuff✔️ Missing Values✔️ All The Numerical Variables✔️ Distribution of the Numerical Variables✔️ Categorical Variables✔️ Cardinality of Categorical Variables✔️ Finding outliers✔️ Relationship between independent and dependent feature  Missing Values<jupyter_code>missing_num = df[df.columns].isna().sum().sort_values(ascending=False)

# total missing values in data frame
missing_num<jupyter_output><empty_output><jupyter_text>Since there are no missing values , this data frame is pretty much clean. So we don't need to deal with missing values in Feature Engineering section  Numerical and Categorical features/ Variables<jupyter_code>numerical_cols = [col for col in df.columns if df[col].dtype!='O']
categorical_cols = [col for col in df.columns if df[col].dtype=='O']
print(f"Numberical columns are = {numerical_cols}")
print(f"Categorical columns are = {categorical_cols}")
continuous_cols = [col for col in numerical_cols if df[col].nunique() >= 25]
discrete_cols = [col for col in numerical_cols if df[col].nunique() < 25]
print(f"Continuous values features are = {continuous_cols}")
print(f"Discrete values feature is = {discrete_cols}")<jupyter_output>Continuous values features are = []
Discrete values feature is = ['Age', 'Internships', 'CGPA', 'Hostel', 'HistoryOfBacklogs', 'PlacedOrNot']<jupyter_text>*There are two types of Numerical variables :* * Discrete Variables * Continuous Variable*But in this data frame all the feature comes in Discrete Variables beacause there are less than 25 unique values in numerical features. So we we put all togather in Numerical feature*   Distribution of the Numerical Variables<jupyter_code>plt.figure(figsize=(12, 14))

for (i, col) in enumerate(discrete_cols,1):
    plt.subplot(6, 2, i)
    sns.histplot(df[col], bins=25)

plt.tight_layout()
plt.show()
plt.figure(figsize=(18, 16))

for (i, col) in enumerate(numerical_cols[:-1],1):
    plt.subplot(6, 2, i)
    sns.histplot(x=df[col], kde=True)

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Categorical VariablesThe dataset contains only two categorical features Gender and Stream.<jupyter_code>plt.figure(figsize=(10, 6))

sns.countplot(y = df['Gender'])<jupyter_output><empty_output><jupyter_text>Here we can see that Number of male students are much much heigher than female students.<jupyter_code>plt.figure(figsize=(10, 6))

sns.countplot(y = df['Stream'])<jupyter_output><empty_output><jupyter_text>Here we can see that Computer Science stream is most popular among students  outliers<jupyter_code>plt.figure(figsize=(14, 8))

count = 1
for col in numerical_cols:
    plt.subplot(4, 2, count)
    sns.boxplot(x=df[col])
    count += 1

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Hypothesis 1. Gender Vs placement___Here we will view the realtionship between gender and placement, whether being male or female can change their placement possibility or not.In reality it should not show any kind of difference because of gender-eqality<jupyter_code>plt.figure(figsize=(10, 6))

sns.countplot(y = df['PlacedOrNot'], hue=df['Gender'])
plt.show()
df['Gender'].value_counts()
total_no_males = df['Gender'].value_counts()['Male']
total_no_females = df['Gender'].value_counts()['Female']

print(f"Number of male students = {total_no_males} and female students = {total_no_females}")

filt = df['PlacedOrNot'] == 1
placed_males = df[filt]['Gender'].value_counts()['Male']
placed_females = df[filt]['Gender'].value_counts()['Female']

print(f"Number students who were placed male = {placed_males} and female = {placed_females}")
print(f"Percentage of beeing placed among males = {placed_males / total_no_males * 100 : 0.2f} %")
print(f"Percentage of beeing placed among males = {placed_females / total_no_females * 100 : 0.2f} %")<jupyter_output>Number of male students = 2475 and female students = 491
Number students who were placed male = 1364 and female = 275
Percentage of beeing placed among males =  55.11 %
Percentage of beeing placed among males =  56.01 %<jupyter_text>So as per our observation it is following our H0 with same percentage of placement irrespective of their gender. 2. CGPA Vs placement___Here We are checking if a student's CGPA is heigher than avg then can it helps in placement timesOur H0 hypothesis is it can not help. Meaning depending on someones CGPA it should not matter whether he/she be placed or not.<jupyter_code>filt = df['CGPA'] > df['CGPA'].mean()

df_heigher_cgpa = df[filt]

plt.figure(figsize=(10, 6))

sns.countplot(y = df_heigher_cgpa['PlacedOrNot'], hue=df_heigher_cgpa['CGPA'])
plt.show()
filt.value_counts()
df_lower_cgpa = df[~filt]
plt.figure(figsize=(10, 6))

sns.countplot(y = df_lower_cgpa['CGPA'], hue=df_lower_cgpa['PlacedOrNot'])
plt.show()<jupyter_output><empty_output><jupyter_text>As we are seeing in the plots, there is no one who haven't get placed with more than average CGPA.And in case of lower CGPA than the average, there is huge difference between number of student have got placement and who havn't. 3. Internship Vs placement___Here We are checking if a student have any internship before their 6th sem can it effect their placementOur H0 hypothesis is it can not have any effect. Meaning if a student have 0 internship and others who have multiple internships should be equally be juged.<jupyter_code>df_internships = df.copy()
df_internships['Internships'] = np.where(df['Internships'] >= 1, 'yes', 'no')
df_internships['PlacedOrNot'] = np.where(df['PlacedOrNot'] == 1, 'placed', 'not placed')
plt.figure(figsize=(10, 6))

sns.countplot(y = df_internships['Internships'], hue=df_internships['PlacedOrNot'])
plt.show()
df_internships = df.copy()
df_internships['Internships'] = np.where(df['Internships'] >= 2, 'yes', 'no')
df_internships['PlacedOrNot'] = np.where(df['PlacedOrNot'] == 1, 'placed', 'not placed')
plt.figure(figsize=(10, 6))

sns.countplot(y = df_internships['Internships'], hue=df_internships['PlacedOrNot'])
plt.show()<jupyter_output><empty_output><jupyter_text>We are not getting any idea by plotting the graph between 'No internship taker' and 'At least one internship taker'.So we plotted a graph between 'zero or one internship taker' and 'atleast two internship taker', it shows a huge improvement in getting placement. 4. Backlog Vs Placement___Here We are checking if a student have any backlog before their 6th sem and can it effect their placementOur H0 hypothesis is it can not have any effect. Meaning if a student have backlogs and others who have no backlog should be equally be juged.<jupyter_code>df_backlog = df.copy()
df_backlog['PlacedOrNot'] = np.where(df['PlacedOrNot'] == 1, 'placed', 'not placed')
df_backlog['HistoryOfBacklogs'] = np.where(df['HistoryOfBacklogs'] >= 1, 'had backlogs', 'had no backlogs')

plt.figure(figsize=(10, 6))

sns.countplot(y = df_backlog['PlacedOrNot'], hue=df_backlog['HistoryOfBacklogs'])
plt.show()<jupyter_output><empty_output><jupyter_text>As per observation, it has no relation between placement and backlog.   5. Stream Vs Placement___Here We are checking if the student have any difference in chance of placement based on their respective branch.Our H0 hypothesis is it should not have effect on the placement.<jupyter_code>df_stream = df.copy()
df_stream['PlacedOrNot'] = np.where(df['PlacedOrNot'] == 1, 'placed', 'not placed')

plt.figure(figsize=(10, 10))

sns.countplot(y = df_stream['Stream'], hue=df_stream['PlacedOrNot'])
plt.show()
df['Stream'].value_counts()
df.groupby(['PlacedOrNot'])['Stream'].value_counts(normalize=True)
df[df['PlacedOrNot']==1]['Stream'].value_counts() / df['Stream'].value_counts() * 100<jupyter_output><empty_output><jupyter_text>So we see, the stream wise percentage is doesn't differ very much than others. 6. Hostel Vs placement___Here We are checking if there is any change in possibility of placement based on Our H0 hypothesis is it can not have any effect. The day scholar and hostel student should have equal chance in placement.<jupyter_code>df_hostel = df.copy()
df_hostel['PlacedOrNot'] = np.where(df['PlacedOrNot'] == 1, 'placed', 'not placed')
df_hostel['Hostel'] = np.where(df['Hostel'] == 1, 'hostel', 'no hostel')

plt.figure(figsize=(10, 10))

sns.countplot(y = df_hostel['Hostel'], hue=df_hostel['PlacedOrNot'])
plt.show()<jupyter_output><empty_output><jupyter_text>So based on whether they live in hostel or not, they are not being discriminated in their placement. 7. Age Vs placement___Here We are checking if a student have any advanges in placement based in their age.Our H0 hypothesis is it can not have any effect. Meaning it should not be discriminated depending based in their age, they should be equally juged.<jupyter_code>df_age = df.copy()

df_age['Age'] = np.where(df_age['Age'] > df_age['Age'].mean(), 'heigher age', 'lower age')
df_hostel['PlacedOrNot'] = np.where(df['PlacedOrNot'] == 1, 'placed', 'not placed')

plt.figure(figsize=(10, 10))

sns.countplot(y = df_hostel['Age'], hue=df_hostel['PlacedOrNot'])
plt.show()

plt.figure(figsize=(10, 6))
sns.histplot(df['Age'], kde= False, bins=10)
plt.axvline(df['Age'].median(), color = '#fc4f30', label = 'Median age', linewidth = 3)
plt.show()<jupyter_output><empty_output><jupyter_text>So depending on their age placement are not being discriminated.<jupyter_code>plt.figure(figsize=(10, 6))
sns.countplot(x=df['Internships'].value_counts(),data=df)<jupyter_output><empty_output><jupyter_text>checking for multi-colinearity<jupyter_code>df.corr()
plt.figure(figsize=(8, 8))
sns.heatmap(df.corr(), annot=True, fmt='0.2f')<jupyter_output><empty_output><jupyter_text>In this plot more common values or higher activities are in brighter colors and less common or activity value are in darker colorsNow we will analysis each numerical feature one by one and find out how it olter the possibality that a student will get placement or notFirst we will check the first numerical feature age. We will count how same age are there<jupyter_code>plt.figure(figsize=(14, 4))
plt.plot(df.corr()['PlacedOrNot'])


def calc_vif(X):
    # Calculating VIF
    vif = pd.DataFrame()
    
    vif["variables"] = X.columns
    vif["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

    return vif

calc_vif(df[numerical_cols[:-1]])<jupyter_output><empty_output><jupyter_text>'CGPA', 'Age' these features are having very high VIF score which is not good. Because high VIF > 10 meaning multi-colinearity exist in data. Multi-Colinearity is very bad thing not only for regression but also for classification. So we will drop these columns and lets see how it affected others VIF score again<jupyter_code>df_cpy = df.drop('CGPA', axis=1)
calc_vif(df_cpy[['Age', 'Internships', 'Hostel', 'HistoryOfBacklogs']])<jupyter_output><empty_output><jupyter_text>So just removing CGPA all others VIF score is now under 10 which is good for our model.  📈 Feature EngineeringIn Feature Engineering We will be handling To Find out the below stuff- Filling Missing values(numerical and categorical)- removing outliers from dataset- transform data to normal distribution(only continuous values)- scaling values(continuous values)- performing categorical encoding(One-Hot encoding)  Filling Missing values____Because the dataset doesn't contain any missing values thats why there is no need to apply any methods  removing outliers from dataset___Only `Age` column containing outliers.<jupyter_code>def plot_data(df,feature):
    plt.figure(figsize=(10, 6))
    
    plt.subplot(1,2,1)
    plt.title(col)
    df[feature].hist()
    
    plt.subplot(1,2,2)
    stat.probplot(df[feature],dist='norm',plot=pylab) # QQ plot
    
    plt.tight_layout()
    plt.show()
df.head()
for col in numerical_cols[:-1]:
    plot_data(df, col)
def remove_outliers_non_normal(df, feature, CONST=1.75):
    plt.figure(figsize=(8, 4))
    newName =  feature + "_copy" 
    
    df[newName] = df[feature].copy()


    plt.subplot(1, 2, 1)
    sns.histplot(df[feature], bins=50)
    plt.title('Before removing outliers')

    # ================================================================
    # removing outliers from non-normal distribution
    IQR = df[newName].quantile(0.75) - df[newName].quantile(0.25)
    lower_boundary = df[newName].quantile(0.25)-(IQR*CONST)
    uppper_boundary = df[newName].quantile(0.75)+(IQR*CONST)
    
    print(df[feature].describe())
    print(f"lower_boundary={lower_boundary} and uppper_boundary={uppper_boundary}")
    
    filt_left = df[newName] <= lower_boundary
    filt_right = df[newName] >= uppper_boundary

    df.loc[filt_left , newName] = lower_boundary
    df.loc[filt_right , newName] = uppper_boundary
    # ================================================================

    plt.subplot(1, 2, 2)
    sns.histplot(df[newName], bins=50)
    plt.title(newName)

    plt.show()
    
    
    plt.figure(figsize=(8, 4))

    plt.subplot(1, 2, 1)
    sns.boxplot(x=df[feature])
    plt.title('before')

    plt.subplot(1, 2, 2)
    sns.boxplot(x=df[newName])
    plt.title('after')

    plt.show()
    
    return newName 
remove_outliers_non_normal(df, 'Age', 1.50)
df['Age'] = df['Age_copy']
df.drop('Age_copy', axis=1, inplace=True)
plot_data(df, 'Age')<jupyter_output><empty_output><jupyter_text>We do not require to transform data to normal distribution because we are going to use XGBoost and XGBoost build a lot decesion tress and prune them.In decision tree do not require to scale any values, So we are also not scaling any variables.  Train Test Splitbefore scaling values i must split it into train and test test to avoid data leakage.  performing categorical encoding(One-Hot encoding)applying one-hot encoding on `'Gender','Stream'` columns<jupyter_code>df = pd.get_dummies(df, columns=['Gender','Stream'], drop_first=True)
df.head()
X = df.drop('PlacedOrNot', axis=1)
y = df['PlacedOrNot']



X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0)
print(f"X_train.shape={X_train.shape}")
print(f"X_test.shape={X_test.shape}")
print(f"y_train.shape={y_train.shape}")
print(f"y_test.shape={y_test.shape}")<jupyter_output>X_train.shape=(1987, 11)
X_test.shape=(979, 11)
y_train.shape=(1987,)
y_test.shape=(979,)<jupyter_text>Model Building and Evaluation<jupyter_code>xgb_cl = xgb.XGBClassifier(objective='binary:logistic', seed=0)
xgb_cl.fit(X_train, y_train, verbose=True, eval_metric='auc', eval_set=[(X_test, y_test)],
           early_stopping_rounds=20)
y_pred = xgb_cl.predict(X_test)
print(classification_report(y_test, y_pred))
plot_confusion_matrix(xgb_cl, X_test, y_test, values_format='d',
                      display_labels=['not placed', 'placed'])<jupyter_output><empty_output><jupyter_text>With default prarmeters XGBoost gave us 87% accuracy which is very good, and lets optimize.  Optimizing XGBoost using GridSearchCV<jupyter_code># param_grid = {
#             'max_depth':[3, 4, 5],
#             'learning_rate':[0.01, 0.013, 0.15, 0.2],
#             'gamma':[0, 0.1, 0.25, 0.5],
#             'reg_lamda':[0, 0.25, 0.5, 1],
#             'scale_pos_weight':[1, 3, 5],
#             'subsample': [0.5, 0.75, 0.9]
# }
# kfold = KFold(10)
# optimal_params = GridSearchCV(xgb.XGBClassifier(objective='binary:logistic', missing=None, seed=0, subsample=0.9, colsample_bytree=0.5),
#                             param_grid=param_grid,
#                             verbose=0,
#                             n_jobs=-1,
#                             cv=5)
# optimal_params.get_params
# optimal_params.fit(X_train, y_train)
# print(optimal_params.best_score_)
# print(optimal_params.best_params_)
# print(optimal_params.best_estimator_)
xgb_cl = xgb.XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=0.5, gamma=0, gpu_id=-1,
              importance_type='gain', interaction_constraints='',
              learning_rate=0.01, max_delta_step=0, max_depth=3,
              min_child_weight=1, missing=1, monotone_constraints='()',
              n_estimators=100, n_jobs=4, num_parallel_tree=1, random_state=0,
              reg_alpha=0, reg_lambda=1, reg_lamda=0, scale_pos_weight=1,
              seed=0, subsample=0.5, tree_method='exact', validate_parameters=1,
              verbosity=None)

xgb_cl.fit(X_train, y_train, verbose=True, eval_metric='auc', eval_set=[(X_test, y_test)],
           early_stopping_rounds=20)
y_pred = xgb_cl.predict(X_test)
print(classification_report(y_test, y_pred))
plot_confusion_matrix(xgb_cl, X_test, y_test, values_format='d',
                      display_labels=['not placed', 'placed'])<jupyter_output>precision    recall  f1-score   support

           0       0.79      0.97      0.87       426
           1       0.97      0.81      0.88       553

    accuracy                           0.88       979
   macro avg       0.88      0.89      0.88       979
weighted avg       0.90      0.88      0.88       979<jupyter_text>So after using gridSearchCV our model did not perform any better.  Saving Model<jupyter_code>model_name = 'xgboost_clasifier.sav'
model = xgb_cl
joblib.dump(xgb_cl, fr'model/{model_name}')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11937938531475442, 'fraction_numerical': 0.019764798893171263, 'mean_word_length': 5.221641561635414, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 0, '>': 111, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '179638', 'license': 'MIT', 'n_tokens_mistral': 7008, 'n_tokens_neox': 6536, 'n_words': 1916, 'path': 'jNotebook Files/Engineering_Placements_Prediction.ipynb', 'repo_name': 'Amit-Kumar-Mondal-Tech/Engineering-Placements-Prediction', 'size': '20238'}