<jupyter_start><jupyter_text>Advance Data Science And Architecture Assignment  1 Machine Learning Models Name: - <NAME>---NUID: - 001059338 **Abstract: -** Fraudulent job postings are everywhere and they exist for various reasons. Some employers post fake jobs to gauge the current talent pool, some people use fake job postings for resume plagiarism, and some job postings are simply spam. As job seekers, we do not want to waste our time and energy on fake job postings. Therefore, the aim of this notebook is to classify whether the job posted is real or fraud using machine learning models. Dataset: - The Employment Scam Aegean Dataset (EMSCAD) is a publicly available dataset containing 17,880 real-life job ads that aims at providing a clear picture of the Employment Scam problem to the research community and can act as a valuable testbed for scientists working on the field. EMSCAD records were manually annotated and classified into two categories. More specifically, the dataset contains 17,014 legitimate and 866 fraudulent job ads published between 2012 to 2014. Dependencies: -<jupyter_code>pip install numpy
pip install pandas
pip install matplotlib
pip install seaborn
pip install scikit-learn<jupyter_output><empty_output><jupyter_text>Dependencies for scikit-learn Python (>= 3.6)NumPy (>= 1.13.3)SciPy (>= 0.19.1)joblib (>= 0.11)threadpoolctl (>= 2.0.0) Importing Libraries and Dataset<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
data = pd.read_csv('https://raw.githubusercontent.com/sumedhtelang/Fake_job_posting_classification/master/data/fake_job_postings.csv')
data.shape
#get a list of the column headers and the datatypes in each column
data.dtypes<jupyter_output><empty_output><jupyter_text>Identifying problems in the dataset<jupyter_code>#Checking for and remove duplicated data in the dataset

duplicates = data.drop_duplicates(keep="first")
diff = data.shape[0] - duplicates.shape[0]
print(f"There were number of duplicate records identified and removed in the data set are {diff}")
#Identifying number of missing values in the data set

data.isnull().sum()
#Column "salary range" is > 80% null that’s why dropping the column

data = data.drop('salary_range', axis=1)<jupyter_output><empty_output><jupyter_text>The columns in this data set are mostly categorical, and have a lot of missing values in some of the columns. Because of that I need to decide how to handle the missing values. Without analyzing every row in the data set, it looks like the missing values are MAR and not MNAR (we expect not all job posters provide all the data points), in this case I am going to take the approach of bucketing (replace) the missing values depending on the type of data stored in the column<jupyter_code>data = data.fillna('NA')
data.isnull().sum()
data.describe()<jupyter_output><empty_output><jupyter_text>Exploratry Data Analysis **Number of characters: -** Let's compare the number of character in the fake post and real post and try to distinguish pattern in the fake and real post based on number of charater used in the post.<jupyter_code>cat_cols = ["telecommuting", "has_company_logo", "has_questions", "employment_type", "required_experience", "required_education",]
# visualizating catagorical variable by target
import matplotlib.gridspec as gridspec # to do the grid of plots
grid = gridspec.GridSpec(3, 3, wspace=0.5, hspace=0.5) # The grid of chart
plt.figure(figsize=(15,25)) # size of figure

# loop to get column and the count of plots
for n, col in enumerate(data[cat_cols]): 
    ax = plt.subplot(grid[n]) # feeding the figure of grid
    sns.countplot(x=col, data=data, hue='fraudulent', palette='Set2') 
    ax.set_ylabel('Count', fontsize=12) # y axis label
    ax.set_title(f'{col} Distribution by Target', fontsize=15) # title label
    ax.set_xlabel(f'{col} values', fontsize=12) # x axis label
    xlabels = ax.get_xticklabels() 
    ylabels = ax.get_yticklabels() 
    ax.set_xticklabels(xlabels,  fontsize=10)
    ax.set_yticklabels(ylabels,  fontsize=10)
    plt.legend(fontsize=8)
    plt.xticks(rotation=90) 
    total = len(data)
    sizes=[] # Get highest values in y
    for p in ax.patches: # loop to all objects
        height = p.get_height()
        sizes.append(height)
        ax.text(p.get_x()+p.get_width()/2.,
                height + 3,
                '{:1.2f}%'.format(height/total*100),
                ha="center", fontsize=10) 
    ax.set_ylim(0, max(sizes) * 1.15) #set y limit based on highest heights


plt.show()<jupyter_output>/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:17: UserWarning: FixedFormatter should only be used together with FixedLocator
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:17: UserWarning: FixedFormatter should only be used together with FixedLocator
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:17: UserWarning: FixedFormatter should only be used together with FixedLocator
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:17: UserWarning: FixedFormatter should only be used together with FixedLocator
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:17: UserWarning: FixedFormatter should only be used together with FixedLocator
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:17: U[...]<jupyter_text>Company profile: - We can see that fake post has less characters in the company profile while real post has more charaters and pattern of words in company profile is same as character in company profile. fake post has less words in the company profile while real post has more words.<jupyter_code>text_df = data[["title", "company_profile", "description", "requirements", "benefits","fraudulent"]]
fig,(ax1,ax2)= plt.subplots(ncols=2, figsize=(17, 5), dpi=100)
length=text_df[text_df["fraudulent"]==1]['company_profile'].str.len()
ax1.hist(length,bins = 20,color='orangered')
ax1.set_title('Fake Post')
length=text_df[text_df["fraudulent"]==0]['company_profile'].str.len()
ax2.hist(length, bins = 20)
ax2.set_title('Real Post')
fig.suptitle('Characters in Company Profile')
plt.show()
fig,(ax1,ax2)= plt.subplots(ncols=2, figsize=(17, 5), dpi=100)
num=text_df[text_df["fraudulent"]==1]['company_profile'].str.split().map(lambda x: len(x))
ax1.hist(num,bins = 20,color='orangered')
ax1.set_title('Fake Post')
num=text_df[text_df["fraudulent"]==0]['company_profile'].str.split().map(lambda x: len(x))
ax2.hist(num, bins = 20)
ax2.set_title('Real Post')
fig.suptitle('Words in company profile')
plt.show()<jupyter_output><empty_output><jupyter_text>Description: - The distribution of charaters in description of the fake and real post are similar but some fake post reach to 6000 to 6500 characters and Both the post has similar distribution of words in description.<jupyter_code>fig,(ax1,ax2)= plt.subplots(ncols=2, figsize=(17, 5), dpi=100)
length=text_df[text_df["fraudulent"]==1]['requirements'].str.len()
ax1.hist(length,bins = 20,color='orangered')
ax1.set_title('Fake Post')
length=text_df[text_df["fraudulent"]==0]['requirements'].str.len()
ax2.hist(length, bins = 20)
ax2.set_title('Real Post')
fig.suptitle('Characters in description')
plt.show()
fig,(ax1,ax2)= plt.subplots(ncols=2, figsize=(17, 5), dpi=100)
num=text_df[text_df["fraudulent"]==1]['description'].str.split().map(lambda x: len(x))
ax1.hist(num,bins = 20,color='orangered')
ax1.set_title('Fake Post')
num=text_df[text_df["fraudulent"]==0]['description'].str.split().map(lambda x: len(x))
ax2.hist(num, bins = 20)
ax2.set_title('Real Post')
fig.suptitle('Words in description')
plt.show()<jupyter_output><empty_output><jupyter_text>Correlation Heatmap: -<jupyter_code>correlation = data.corr() 
f, ax = plt.subplots(figsize= (20,20))
sns.heatmap(correlation)
plt.xticks(fontsize = 23)
plt.yticks(fontsize = 23)
plt.show()<jupyter_output><empty_output><jupyter_text>Building Machine Learning Models Building various classifier models and then attempt to tune them to achieve higher levels of accuracy when identifying the fradulent job postings. Next split up this data set into train and test to allow the model to work on unseen data. Later doing a validation holdout on the data and plot the performance of the test vs. validation results to try and ensure my models aren't overfitting.<jupyter_code>#Before converting categorical variables we have to drop the free text columns that can't be converted
#also dropping the job id column since it's an id field

data = data.drop(['job_id','company_profile','description','benefits','requirements'],axis=1)
#I am using pd.get_dummies to convert the categorical attributes to boolean columns to be used in the models
#Get dummies will handle creation of the new columns representing possible categorical values as numerical 1/0

data = pd.get_dummies(data, prefix_sep = '_', drop_first=True)
data.shape
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.datasets import make_classification
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score
# Dropping class label from features X
X = data.drop('fraudulent',axis=1)

# Target class label assigned to y
y = data['fraudulent']
# 70/30 split of train/test data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=1)<jupyter_output><empty_output><jupyter_text>Random Forest classifier<jupyter_code>from sklearn.ensemble import RandomForestClassifier 
from sklearn.metrics import confusion_matrix,classification_report
from time import time
rfc = RandomForestClassifier() # define the model 

start = time()
#fit the classifier to the training data 
rfc.fit(X_train, y_train)

print(f"Accuracy: {round(rfc.score(X_test, y_test)*100)}%")

print("Parameters: ", rfc.get_params())
#print the confusion matrix and classifier scores for the model
from sklearn.metrics import plot_confusion_matrix
y_pred = rfc.predict(X_test)
randomcm = confusion_matrix(y_test, y_pred)
plot_confusion_matrix(rfc, X_test, y_test)  # doctest: +SK IP

print(classification_report(y_test, y_pred))
print(randomcm)
# plot the ROC curve and calculate the AUC
# create a classifier that always picks 0 choice (to compare result to the classifier model)
ns_probs = [0 for _ in range(len(y_test))]
lr_probs = rfc.predict_proba(X_test)

# keep probabilities for the positive outcome only
lr_probs = lr_probs[:, 1]

# calculate scores
ns_auc = roc_auc_score(y_test, ns_probs)
lr_auc = roc_auc_score(y_test, lr_probs)

# summarize scores
print('0 Classifier: ROC AUC=%.3f' % (ns_auc))
print('RFC Classifier: ROC AUC=%.3f' % (lr_auc))

# calculate roc curves
ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)
lr_fpr, lr_tpr, _ = roc_curve(y_test, lr_probs)

# plot the roc curve for the model
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='0 Classifier')
plt.plot(lr_fpr, lr_tpr, marker='.', label='RFC Classifier')

# axis labels
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')

# show the legend
plt.legend()

# show the plot
plt.show()<jupyter_output>0 Classifier: ROC AUC=0.500
RFC Classifier: ROC AUC=0.954<jupyter_text>To get the relative feature importance for the features used by the RFC. Printing this out I can see that 'has_company-logo' had the highest importance in this model; most fraud was associated with no logo. We also see Administrative positions as having somewhat importance; which also aligns with the correlations seen earlier.<jupyter_code>#obtain the relative feature importance of the rf classifier
feature_imp = pd.DataFrame(rfc.feature_importances_, index=X_train.columns,
columns=['importance']).sort_values('importance', ascending=False)
feature_imp<jupyter_output><empty_output><jupyter_text>Gaussian Naive Bayes Classifier<jupyter_code>from sklearn.naive_bayes import GaussianNB
nb = GaussianNB()

start = time()
#fit the classifier to the training data 
nb.fit(X_train, y_train)
print("Naieve Bayes Classifier took %.2f seconds"
      % (time() - start,))

y_pred = nb.predict(X_test)
print(f"Accuracy: {round(nb.score(X_test, y_test)*100)}%")
print("Parameters: ", nb.get_params())
#print the confusion matrix and classifier scores for the model
from sklearn.metrics import plot_confusion_matrix
y_pred = nb.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
plot_confusion_matrix(nb, X_test, y_test)  # doctest: +SK IP

print(classification_report(y_test, y_pred))
print(cm)
# plot the ROC curve and calculate the AUC
# create a classifier that always picks 0 choice (to compare result to the classifier model)
ns_probs = [0 for _ in range(len(y_test))]
lr_probs = nb.predict_proba(X_test)

# keep probabilities for the positive outcome only
lr_probs = lr_probs[:, 1]

# calculate scores
ns_auc = roc_auc_score(y_test, ns_probs)
lr_auc = roc_auc_score(y_test, lr_probs)

# summarize scores
print('0 Classifier: ROC AUC=%.3f' % (ns_auc))
print('NB Classifier: ROC AUC=%.3f' % (lr_auc))

# calculate roc curves
ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)
lr_fpr, lr_tpr, _ = roc_curve(y_test, lr_probs)

# plot the roc curve for the model
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='0 Classifier')
plt.plot(lr_fpr, lr_tpr, marker='.', label='NB Classifier')

# axis labels
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')

# show the legend
plt.legend()

# show the plot
plt.show()<jupyter_output>0 Classifier: ROC AUC=0.500
NB Classifier: ROC AUC=0.831<jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
lreg = LogisticRegression()
start = time()
#fit the classifier to the training data 
lreg.fit(X_train, y_train)
print("Logistic Regression Classifier took %.2f seconds"
      % (time() - start,))

y_pred = lreg.predict(X_test)
print(f"Accuracy: {round(lreg.score(X_test, y_test)*100)}%")
print("Parameters: ", lreg.get_params())
#print the confusion matrix and classifier scores for the model
from sklearn.metrics import plot_confusion_matrix
y_pred = lreg.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
plot_confusion_matrix(nb, X_test, y_test)  # doctest: +SKIP

print(classification_report(y_test, y_pred))
print(cm)
# plot the ROC curve and calculate the AUC
# create a classifier that always picks 0 choice (to compare result to the classifier model)
ns_probs = [0 for _ in range(len(y_test))]
lr_probs = lreg.predict_proba(X_test)

# keep probabilities for the positive outcome only
lr_probs = lr_probs[:, 1]

# calculate scores
ns_auc = roc_auc_score(y_test, ns_probs)
lr_auc = roc_auc_score(y_test, lr_probs)

# summarize scores
print('0 Classifier: ROC AUC=%.3f' % (ns_auc))
print('NB Classifier: ROC AUC=%.3f' % (lr_auc))

# calculate roc curves
ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)
lr_fpr, lr_tpr, _ = roc_curve(y_test, lr_probs)

# plot the roc curve for the model
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='0 Classifier')
plt.plot(lr_fpr, lr_tpr, marker='.', label='NB Classifier')

# axis labels
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')

# show the legend
plt.legend()

# show the plot
plt.show()
importance = lreg.coef_[0]

feature_imp = pd.DataFrame(-importance, index=X_train.columns,
columns=['importance']).sort_values('importance', ascending=False)
feature_imp<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10720010258383023, 'fraction_numerical': 0.017952170289158173, 'mean_word_length': 5.972731336611533, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '653459', 'license': 'MIT', 'n_tokens_mistral': 5447, 'n_tokens_neox': 5092, 'n_words': 1546, 'path': 'Fake_job_posting_classification.ipynb', 'repo_name': 'sumedhtelang/Fake_job_posting_classification', 'size': '15616'}