<jupyter_start><jupyter_text><jupyter_code># This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt
import seaborn as sns


# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session<jupyter_output>/kaggle/input/hr-analytics-job-change-of-data-scientists/sample_submission.csv
/kaggle/input/hr-analytics-job-change-of-data-scientists/aug_test.csv
/kaggle/input/hr-analytics-job-change-of-data-scientists/aug_train.csv<jupyter_text>OverviewThis notebook is organized as follows :1. Reading the data2. EDA - Exploratory data analysis3. Data preproccesing and cleaning4. Training models5. Results  1. Reading the data<jupyter_code>train = pd.read_csv('../input/hr-analytics-job-change-of-data-scientists/aug_train.csv')
test = pd.read_csv('../input/hr-analytics-job-change-of-data-scientists/aug_test.csv')<jupyter_output><empty_output><jupyter_text>2. EDA - Exploratory data analysis Explore the data.First I visualized the relationships between each variable to the other variables in the data using Pairplot. Then I used methods like head(), describe(), info() ect. to display the first five rows of each data set, display basic statistical details and general information about the data.Afterwards I visualized the missing data using the missingno library in order to decide which colunmns to drop and how to deal with the missing values, together with the pairplot and other visualizations shown below.  Train set<jupyter_code>train.head()
train.shape
train.columns
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 19158 entries, 0 to 19157
Data columns (total 14 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   enrollee_id             19158 non-null  int64  
 1   city                    19158 non-null  object 
 2   city_development_index  19158 non-null  float64
 3   gender                  14650 non-null  object 
 4   relevent_experience     19158 non-null  object 
 5   enrolled_university     18772 non-null  object 
 6   education_level         18698 non-null  object 
 7   major_discipline        16345 non-null  object 
 8   experience              19093 non-null  object 
 9   company_size            13220 non-null  object 
 10  company_type            13018 non-null  object 
 11  last_new_job            18735 non-null  object 
 12  training_hours          19158 non-null  int64  
 13  target                  19158 non-null  float64
dtypes: float64(2), int64(2), object(10)
me[...]<jupyter_text>As shown above, this data set contains 14 columns. i.e 13 features + 1 label.10 categorical variables: `city`, `gender`, `relevent_experience`, `enrolled_university`, `education_level`, `major_discipline`, `experience`, `company_size`,`last_new_job`,`company_type`3 numerical variables: `enrollee_id`, `city_development_index`,`training_hours`1 label variable - `target`<jupyter_code>train.describe().transpose()<jupyter_output><empty_output><jupyter_text>Visualizing the data<jupyter_code>import seaborn as sns

sns.pairplot(train, hue='target')
train.isnull().sum()
# Visualize missing data 

import missingno as msno
import matplotlib.pyplot as plt
msno.matrix(train)
plt.show()<jupyter_output><empty_output><jupyter_text>White lines indicate missing values, while black lines represent existing information.As one can see, the `gender`, `major_discipline`, `company_size` and `company_type` columns have many missing value, scattered throughout the columns, and the `enrolled_university`, `education_level`,`experience` and `last_new_job`  column has only few missing values.<jupyter_code>msno.heatmap(train)<jupyter_output><empty_output><jupyter_text>The missingno correlation heatmap measures nullity correlation, i.e how strongly the presence or absence of one variable affects the presence of another ([link for more information](https://github.com/ResidentMario/missingno))In this example, it seems that if the variable `company_size` appears, than the variable `company_type` is also likely to appear (correlation = 0.8)  Test set<jupyter_code>test.head()
test.shape
test.info()
test.isnull().sum()
# Visualize missing data in the test data

msno.matrix(test)
plt.show()<jupyter_output><empty_output><jupyter_text>The same conclusions for the test set - most of the `gender`,`company_size` and `comapny_type` columns are missing, as well as few values in the `enrolled_university`,`education_level`, `experience` and`last_new_job` columns.  3. Data preproccesing and cleaning Create a new dataframe for the rest of the processing, keeping the `enrollee_id` column in the full dataframe for the final stage of writing to the csv file.<jupyter_code>train_df = train.drop(['enrollee_id'], axis=1)
test_df = test.drop(['enrollee_id'], axis=1)<jupyter_output><empty_output><jupyter_text>Dealing with missing values and cleaning the data These data contains ordinal and nominal columns. i.e in some columns the order of the categiries is significnt (ordinal) and in others there is no meaning to the order or the quantity of the value (nominal). The ordinal columns in our data are : `relevent_experience`, `enrolled_university`, `education_level`, `experience`, `company_size`,`last_new_job`,The nominal columns are: `city`, `gender`,  `major_discipline`, `company_type`.  Label EncodingMost machine learning models require the categorical variables to be in one-hot encoding representation, or to be mapped into numbers.we will apply ordinal encoding to the ordinal columns (i.e mapping) and one-hot encoding to the nominal columns.<jupyter_code>def calc_num_categories(column_name):
    unique_values_train = train[column_name].unique()
    number_of_categories_train = len(unique_values_train)

    unique_values_test = test[column_name].unique()
    number_of_categories_test = len(unique_values_test)

    if number_of_categories_train != number_of_categories_test:
        print(f'Unique values train: {unique_values_train}, \n Number of categories: {number_of_categories_train} \n Unique values test: {unique_values_test}, \n Number of categories: {number_of_categories_test}')
    else:
        print(f'Unique values: {unique_values_train}, \n Number of categories: {number_of_categories_train}')

def calc_percent_missing_values(train_df, test_df, column_name):  
    train_missing = train_df[column_name].isnull().sum()
    test_missing = test_df[column_name].isnull().sum()
    perc_train = (train_missing/len(train_df[column_name]))
    perc_test = (test_missing/len(test_df[column_name]))
    
    print(f'{column_name}:\n % of missing values in the train data: {train_missing} \n % of missing values in the test data: {test_missing}\
    \n %train" {perc_train:.2%} \n %test: {perc_test:.2%}')<jupyter_output><empty_output><jupyter_text>1. `experience`<jupyter_code>calc_num_categories('experience')
import matplotlib.pyplot as plt
import seaborn as sns

g = sns.catplot(x="experience", kind="count", data=train, height=8, aspect=10/6, palette="husl",hue='target',
                order= ['<1', '1','2','3','4', '5','6','7','8','9','10', '11','12','13','14','15','16','17','18','19','20','>20'])
calc_percent_missing_values(train, test, 'experience')<jupyter_output>experience:
 % of missing values in the train data: 65 
 % of missing values in the test data: 5    
 %train" 0.34% 
 %test: 0.23%<jupyter_text>Since the 'experience' column is a categorical variable and there are only few missing values in this column, an appropriate choice would be to replace the missing values by the most frequent value of that column.<jupyter_code>train_df["experience"] = train_df["experience"].transform(lambda x: x.fillna(x.mode()[0]))
test_df["experience"] = test_df["experience"].transform(lambda x: x.fillna(x.mode()[0]))
calc_percent_missing_values(train_df, test_df, 'experience')
train_df['experience'].replace({'<1': '0', '>20': '21'}, inplace=True)
test_df['experience'].replace({'<1': '0', '>20': '21'}, inplace=True)
train_df['experience'].unique()
train_df['experience'] = train_df['experience'].astype(int)
test_df['experience'] = test_df['experience'].astype(int)
experience_bin_labels = [1, 2, 3,4]
train_df['Binned_experience'] =   pd.qcut(train_df['experience'], q=4, labels=experience_bin_labels).astype(int)
test_df['Binned_experience'] =    pd.qcut(test_df['experience'], q=4, labels=experience_bin_labels).astype(int)
#sns.set(rc = {'figure.figsize':(12,6)})
plt.figure(figsize = (12,6))
sns.countplot(x="Binned_experience", hue="target", data=train_df, palette="husl")<jupyter_output><empty_output><jupyter_text>2.`last_new_job`<jupyter_code>calc_num_categories('last_new_job')
g = sns.catplot(x="last_new_job", kind="count", data=train, height=8, palette="husl",hue='target', order= ['1','2','3','4','>4', 'never'],aspect=10/6)
calc_percent_missing_values(train, test, 'last_new_job')<jupyter_output>last_new_job:
 % of missing values in the train data: 423 
 % of missing values in the test data: 40    
 %train" 2.21% 
 %test: 1.88%<jupyter_text>Similarly, since the 'last_new_job' column is a categorical variable and there are only few missing values in this column, an appropriate choice would be to replace the missing values by the most frequent value of that column.<jupyter_code>train_df["last_new_job"] = train_df["last_new_job"].transform(lambda x: x.fillna(x.mode()[0]))
test_df["last_new_job"] = test_df["last_new_job"].transform(lambda x: x.fillna(x.mode()[0]))
calc_percent_missing_values(train_df, test_df, 'last_new_job')
from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
train_df['last_new_job'] = le.fit_transform(train_df['last_new_job'])
print(le.classes_)
test_df['last_new_job'] = le.fit_transform(test_df['last_new_job'])<jupyter_output>['1' '2' '3' '4' '>4' 'never']<jupyter_text>3. `enrolled_university`<jupyter_code>calc_num_categories('enrolled_university')
g = sns.catplot(x="enrolled_university", kind="count", data=train, height=5, aspect=1.5, palette="husl",hue='target',order= ['no_enrollment','Part time course','Full time course'])
calc_percent_missing_values(train_df, test_df, 'enrolled_university')<jupyter_output>enrolled_university:
 % of missing values in the train data: 386 
 % of missing values in the test data: 31    
 %train" 2.01% 
 %test: 1.46%<jupyter_text>Similarly, we replace the missing values by the most frequent value of that column.<jupyter_code>train_df["enrolled_university"] = train_df["enrolled_university"].transform(lambda x: x.fillna(x.mode()[0]))
test_df["enrolled_university"] = test_df["enrolled_university"].transform(lambda x: x.fillna(x.mode()[0]))
calc_percent_missing_values(train_df, test_df, 'enrolled_university')
enrolled_university_mapping = {'Full time course': 2, 'Part time course': 1, 'no_enrollment': 0}

train_df['enrolled_university'] = train_df['enrolled_university'].map(enrolled_university_mapping)
test_df['enrolled_university'] = test_df['enrolled_university'].map(enrolled_university_mapping)   

train_df['enrolled_university'].unique()<jupyter_output><empty_output><jupyter_text>4. `education_level`<jupyter_code>calc_num_categories('education_level')
g = sns.catplot(x="education_level", kind="count", data=train_df, height=6, aspect=2, palette="husl",hue='target',order= ['Primary School','High School','Graduate','Masters','Phd'])
calc_percent_missing_values(train_df, test_df, 'education_level')<jupyter_output>education_level:
 % of missing values in the train data: 460 
 % of missing values in the test data: 52    
 %train" 2.40% 
 %test: 2.44%<jupyter_text>Similarly, we replace the missing values by the most frequent value of that column.<jupyter_code>train_df["education_level"] = train_df["education_level"].transform(lambda x: x.fillna(x.mode()[0]))
test_df["education_level"] = test_df["education_level"].transform(lambda x: x.fillna(x.mode()[0]))
calc_percent_missing_values(train_df, test_df, 'education_level')
education_level_mapping = {'Primary School': 1,'High School': 2,'Graduate': 3,'Masters': 4,'Phd': 5}

train_df['education_level'] = train_df['education_level'].map(education_level_mapping)
test_df['education_level'] = test_df['education_level'].map(education_level_mapping)   
train_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>5. `city`<jupyter_code>calc_num_categories('city')<jupyter_output>Unique values train: ['city_103' 'city_40' 'city_21' 'city_115' 'city_162' 'city_176'
 'city_160' 'city_46' 'city_61' 'city_114' 'city_13' 'city_159' 'city_102'
 'city_67' 'city_100' 'city_16' 'city_71' 'city_104' 'city_64' 'city_101'
 'city_83' 'city_105' 'city_73' 'city_75' 'city_41' 'city_11' 'city_93'
 'city_90' 'city_36' 'city_20' 'city_57' 'city_152' 'city_19' 'city_65'
 'city_74' 'city_173' 'city_136' 'city_98' 'city_97' 'city_50' 'city_138'
 'city_82' 'city_157' 'city_89' 'city_150' 'city_70' 'city_175' 'city_94'
 'city_28' 'city_59' 'city_165' 'city_145' 'city_142' 'city_26' 'city_12'
 'city_37' 'city_43' 'city_116' 'city_23' 'city_99' 'city_149' 'city_10'
 'city_45' 'city_80' 'city_128' 'city_158' 'city_123' 'city_7' 'city_72'
 'city_106' 'city_143' 'city_78' 'city_109' 'city_24' 'city_134' 'city_48'
 'city_144' 'city_91' 'city_146' 'city_133' 'city_126' 'city_118' 'city_9'
 'city_167' 'city_27' 'city_84' 'city_54' 'city_39' 'city_79' 'city_76'
 'city_77' 'city_81' 'city_131'[...]<jupyter_text>removing unnecessary text<jupyter_code>train_df['city'] = train_df['city'].str.replace('city_','')
test_df['city'] = test_df['city'].str.replace('city_','')
train_df.city.isnull().values.any()
le = LabelEncoder()
train_df['city'] = le.fit_transform(train_df['city'])
test_df['city'] = le.fit_transform(test_df['city'])<jupyter_output><empty_output><jupyter_text>6. `relevent_experience`<jupyter_code>calc_num_categories('relevent_experience')
g = sns.catplot(x="relevent_experience", kind="count", data=train_df, height=6, aspect=1, palette="husl",hue='target')
relevent_experience_mapping = {'Has relevent experience': 1, 'No relevent experience': 0}

train_df['relevent_experience'] = train_df['relevent_experience'].map(relevent_experience_mapping)
test_df['relevent_experience'] = test_df['relevent_experience'].map(relevent_experience_mapping)   
train_df.relevent_experience.unique()<jupyter_output><empty_output><jupyter_text>7. `gender`<jupyter_code>calc_num_categories('gender')
g = sns.catplot(x="gender", kind="count", data=train_df, height=5, aspect=1.5, palette="husl",hue='target',order= ['Male','Female','Other'])
train_df.gender.unique()
train_df['gender'] = train_df['gender'].fillna('Unknown')
test_df['gender'] = test_df['gender'].fillna('Unknown')
print(train_df.gender.unique())

from sklearn.preprocessing import LabelEncoder

train_df['gender']=train_df['gender'].fillna('Unknown')
test_df['gender'] = test_df['gender'].fillna('Unknown')
print(train_df.gender.unique())

gender_mapping = {'Male': 1,'Female': 2,'Other': 3, 'Unknown': 0}

train_df['gender'] = train_df['gender'].map(gender_mapping)
test_df['gender'] = test_df['gender'].map(gender_mapping)<jupyter_output>['Male' 'Unknown' 'Female' 'Other']<jupyter_text>8. `major_discipline`<jupyter_code>calc_num_categories('major_discipline')
g = sns.catplot(x="major_discipline", kind="count", data=train_df, height=5, aspect=1.8, palette="husl",hue='target')
calc_percent_missing_values(train_df, test_df, 'major_discipline')
train_df['major_discipline']=train_df['major_discipline'].fillna('Unknown')
test_df['major_discipline'] = test_df['major_discipline'].fillna('Unknown')

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder() 
train_df['major_discipline'] = le.fit_transform(train_df['major_discipline'])
print(le.classes_)
test_df['major_discipline'] = le.fit_transform(test_df['major_discipline'])<jupyter_output>['Arts' 'Business Degree' 'Humanities' 'No Major' 'Other' 'STEM' 'Unknown']<jupyter_text>9. `company_size`<jupyter_code>calc_num_categories('company_size')
g = sns.catplot(x="company_size", kind="count", data=train_df, height=5, aspect=1.5, palette="husl",hue='target')
calc_percent_missing_values(train_df, test_df, 'company_size')
train_df['company_size']=train_df['company_size'].fillna('Unknown')
test_df['company_size'] = test_df['company_size'].fillna('Unknown')
print(train_df.company_size.unique())
company_size_mapping = {'<10': 1,'10/49': 2,'50-99': 3, '100-500': 4,'500-999': 5,'1000-4999': 6,'5000-9999': 7,'10000+': 8, 'Unknown': 0}

train_df['company_size'] = train_df['company_size'].map(company_size_mapping)
test_df['company_size'] = test_df['company_size'].map(company_size_mapping)   
print(train_df.company_size.unique())<jupyter_output>[0 3 1 8 7 6 2 4 5]<jupyter_text>10. `company_type`<jupyter_code>calc_num_categories('company_type')
g = sns.catplot(x="company_type", kind="count", data=train_df, height=6, aspect=1.8, palette="husl",hue='target')
calc_percent_missing_values(train_df, test_df, 'company_type')
train_df['company_type']=train_df['company_type'].fillna('Unknown')
test_df['company_type'] = test_df['company_type'].fillna('Unknown')
le = LabelEncoder()
train_df['company_type'] = le.fit_transform(train_df['company_type'])
print(le.classes_)
test_df['company_type'] = le.fit_transform(test_df['company_type'])<jupyter_output>['Early Stage Startup' 'Funded Startup' 'NGO' 'Other' 'Public Sector'
 'Pvt Ltd' 'Unknown']<jupyter_text>Fill missing values using RandomForestClassifier<jupyter_code>train_df.columns

from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

#'city', 'city_development_index', 'relevent_experience', 'enrolled_university', 'education_level','major_discipline', 'experience', 'company_size',\
#'company_type','last_new_job', 'training_hours'

# split the dataset into train and test sets
train_gender = train_df.loc[train_df.gender != 0][['gender','city','experience','city_development_index','major_discipline',\
                                                   'company_size','last_new_job']]
print(train_gender.isnull().sum())
test_gender = train_df.loc[train_df.gender == 0][['gender', 'city', 'experience','city_development_index','major_discipline',\
                                                   'company_size','last_new_job']]


X_train = train_gender.drop('gender', axis=1)
y_train = train_gender.gender
X_test = test_gender.drop('gender', axis=1)
print(X_train.shape)
print(X_test.shape)

#knn = KNeighborsClassifier(n_neighbors=12, p=1)
#knn.fit(X_train, y_train)
#y_pred = knn.predict(X_test)

rf = RandomForestClassifier(n_estimators=2500, n_jobs=-1)
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

train_df.loc[train_df.gender == 0,'gender'] = y_pred<jupyter_output>gender                    0
city                      0
experience                0
city_development_index    0
major_discipline          0
company_size              0
last_new_job              0
dtype: int64
(14650, 6)
(4508, 6)<jupyter_text>PAY ATTENTION to train the model using the train set only. The test set is used for prediction.<jupyter_code>#test set 

test_set_gender = test_df.loc[test_df.gender != 0][['gender','city','experience','city_development_index','major_discipline',\
                                                   'company_size','last_new_job']]
test_set_gender.isnull().sum()
predict_gender = test_df.loc[test_df.gender == 0][['gender','city','experience','city_development_index','major_discipline',\
                                                   'company_size','last_new_job']]


X_test_set = test_set_gender.drop('gender', axis=1)
X_test_set_predict_gender = predict_gender.drop('gender', axis=1)

y_test_pred = rf.predict(X_test_set_predict_gender)
#y_test_pred = knn.predict(X_test_set_predict_gender)

test_df.loc[test_df.gender == 0, 'gender'] = y_test_pred
# split the dataset into train and test sets
train_major_discipline = train_df.loc[train_df.major_discipline != 6][['relevent_experience', 'enrolled_university','major_discipline','training_hours', 'experience',\
                                                   'company_size','company_type','training_hours']]
print(train_major_discipline.isnull().sum())
test_major_discipline = train_df.loc[train_df.major_discipline == 6][['relevent_experience', 'enrolled_university','major_discipline','training_hours', 'experience',\
                                                   'company_size','company_type','training_hours']]


X_train = train_major_discipline.drop('major_discipline', axis=1)
y_train = train_major_discipline.major_discipline
X_test = test_major_discipline.drop('major_discipline', axis=1)
print(X_train.shape)
print(X_test.shape)

#knn = KNeighborsClassifier(n_neighbors=8, p=1)
#knn.fit(X_train, y_train)
#y_pred = knn.predict(X_test)

rf = RandomForestClassifier(n_estimators=2500, n_jobs=-1)
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

train_df.loc[train_df.major_discipline == 6,'major_discipline'] = y_pred
#test set 

test_set_major_discipline = test_df.loc[test_df.major_discipline != 6][['relevent_experience', 'enrolled_university', 'major_discipline','training_hours',\
                                                    'experience','company_size','company_type', 'training_hours']]
test_set_major_discipline.isnull().sum()
predict_major_discipline = test_df.loc[test_df.major_discipline == 6][['relevent_experience', 'enrolled_university','major_discipline','training_hours',\
                                                   'experience','company_size','company_type', 'training_hours']]


X_test_set = test_set_major_discipline.drop('major_discipline', axis=1)
X_test_set_predict_major_discipline = predict_major_discipline.drop('major_discipline', axis=1)

y_test_pred = rf.predict(X_test_set_predict_major_discipline)
#y_test_pred = knn.predict(X_test_set_predict_major_discipline)


test_df.loc[test_df.major_discipline == 6, 'major_discipline'] = y_test_pred<jupyter_output><empty_output><jupyter_text>Label EncodingMost machine learning models require the categorical variables to be in one-hot encoding representation, or to be mapped into numbers.we will apply ordinal encoding to the ordinal columns (i.e mapping) and one-hot encoding to the nominal columns.<jupyter_code>train_df = pd.get_dummies(data=train_df,columns=['company_type','gender','major_discipline'] )
test_df = pd.get_dummies(data=test_df,columns=['company_type','gender','major_discipline'] )
train_df.drop(['experience','city'], axis=1, inplace = True)
test_df.drop(['experience','city'], axis=1,inplace = True)<jupyter_output><empty_output><jupyter_text>4. Training modelsWe will split the data into train and validation sets<jupyter_code># Separate input features and target

y = train_df['target']
X = train_df.drop(['target'], axis=1)
from sklearn.model_selection import train_test_split

# setting up testing and training sets

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size = 0.3, random_state=42)
X_train.shape<jupyter_output><empty_output><jupyter_text>Dealing with the problem of imbalnaced dataImbalanced classes are a common problem in machine learning classification where the ratio of observations in each class is a disproportionate. There are few options for dealing with this problem, such as change the performance metric, change the algorithm, oversample minority class or undersample majority class and generate synthetic samples. here we'll try to use the synthetic data approach. SMOTE generates synthetic data by using the nearest neighbors algorithm.<jupyter_code>from imblearn.over_sampling import SMOTE

sm = SMOTE(random_state=27)
X_train, y_train = sm.fit_resample(X_train, y_train)<jupyter_output><empty_output><jupyter_text>HyperParameter TuningSince grid search tuning can take a long time to run, we don't necessarily want to run it every time.Instead of commenting it out, we use a flag to control whether to run it or not.<jupyter_code>GridSearch = False
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV

if GridSearch == True:
    param_grid = {'n_neighbors': np.arange(1,50), 'p':[1,2]}
    knn = KNeighborsClassifier()
    knn_cv = GridSearchCV(knn, param_grid, cv=10)
    knn_cv.fit(X_train, y_train)
    print(knn_cv.best_params_)
    knn_cv.predict(X_val)
    print(knn_cv.score(X_val, y_val))
    
    #{'n_neighbors': 2, 'p': 1}
    #0.6985038274182325
from sklearn.linear_model import LogisticRegression


if GridSearch == True:
    param_grid = {'C':[0.01, 0.1,0.2,0.25,0.3,0.4, 1, 2],'max_iter': [150,200,250, 300, 1000,1500], 'random_state':[42], 'solver':['liblinear'] } 
    lr = LogisticRegression()
    lr_cv = GridSearchCV(lr, param_grid, cv=10)
    lr_cv.fit(X_train, y_train)
    print(lr_cv.best_params_)
    lr_cv.predict(X_val)
    print(lr_cv.score(X_val, y_val))
    
#{'C': 1, 'max_iter': 150, 'random_state': 42, 'solver': 'liblinear'}
#0.732428670842032
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV


if GridSearch == True:
    param_grid = {'n_estimators':[25,50,60],'max_features': [3,4,5], 'max_depth':[5,6,7], 'min_samples_leaf':[8,9,10,11],'random_state':[1,42] } 
    rf = RandomForestClassifier(n_jobs=-1)
    rf_cv =  GridSearchCV(rf, param_grid)
    rf_cv.fit(X_train, y_train)
    print(rf_cv.best_params_)
    rf_cv.predict(X_val)
    print(rf_cv.score(X_val, y_val))
    
   
# {'max_depth': 7, 'max_features': 5, 'min_samples_leaf': 9, 'n_estimators': 60, 'random_state': 42}
# 0.7299930410577592
from sklearn.linear_model import SGDClassifier

if GridSearch == True:
    param_grid = {'alpha':[0.01, 0.1,0.2, 1, 2],'max_iter': [70,75,80,90,120,150,200], 'random_state':[42] } 
    sgd = SGDClassifier()
    sgd_cv =  GridSearchCV(sgd, param_grid, cv=5)
    sgd_cv.fit(X_train, y_train)
    print(sgd_cv.best_params_)
    sgd_cv.predict(X_val)
    print(sgd_cv.score(X_val, y_val))
    
    #{'alpha': 0.01, 'max_iter': 75, 'random_state': 21}
    # 0.6196938065414057
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

if GridSearch == True:
    param_grid = {'base_estimator': [LogisticRegression(random_state=1,solver='liblinear' ), DecisionTreeClassifier()],'max_features':[3,4,5,6],'max_samples':[30,40,50,60],
                  'n_estimators':[20,30,40,50],'random_state':[1,123, 21,42,10] } 

    bc = BaggingClassifier(n_jobs=-1)
    bc_cv = GridSearchCV(bc, param_grid, cv=5)
    bc_cv.fit(X_train, y_train)
    print(bc_cv.best_params_)
    bc_cv.predict(X_val)
    print(bc_cv.score(X_val, y_val))
    
    #{'base_estimator': DecisionTreeClassifier(), 'max_features': 6, 'max_samples': 60, 'n_estimators': 50, 'random_state': 1}
    # 0.7141614474599861
import xgboost as xgb

if GridSearch == True:
    param_grid = {'n_estimators':[300,500,1000], 'objective':['binary:logistic'],'learning_rate':[0.1,1,1.5],'max_depth':[5,10,15] }
    #xgb_clf = xgb.XGBClassifier(base_score=0.5, objective='binary:logistic', n_estimators=300,seed=1, learning_rate=0.1, max_depth=10)
    xgb = xgb.XGBClassifier()
    xgb_cv =  GridSearchCV(xgb, param_grid, cv=5) 
    xgb_cv.fit(X_train, y_train)
    print(xgb_cv.best_params_)
    xgb_cv.predict(X_val)
    print(xgb_cv.score(X_val,y_val))
    
   # {'learning_rate': 0.1, 'max_depth': 10, 'n_estimators': 300, 'objective': 'binary:logistic'}
   # 0.7632219902574808
import lightgbm as lgb

if GridSearch == True:
   # param_grid = {'boosting_type':['gbdt', 'rf'], 'num_leaves': [10, 20, 31, 40], 'max_depth':[- 1, 1,2,3,5, 10], 'learning_rate':[0.1, 0.01, 1, 10], 
    #                'n_estimators':[50, 100, 150, 200], 'random_state':[42]}
    param_grid = {'boosting_type':['gbdt'], 'num_leaves': [10, 20, 31, 40], 'max_depth':[- 1,1,2,3,5,10], 'learning_rate':[0.1,0.01,1,10], 
                    'n_estimators':[50, 100, 150, 200], 'random_state':[42]}

    LGBM = lgb.LGBMClassifier()
    LGBM_cv = GridSearchCV(LGBM, param_grid, cv=6)
    LGBM_cv.fit(X_train, y_train)
    print(LGBM_cv.best_params_)
    LGBM_cv.predict(X_val)
    print(LGBM_cv.score(X_val, y_val))
    
     #{'boosting_type': 'gbdt', 'learning_rate': 0.1, 'max_depth': 10, 'n_estimators': 200, 'num_leaves': 31, 'random_state': 42}
     # 0.7731384829505915
from sklearn.metrics import accuracy_score, f1_score,recall_score
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import VotingClassifier, RandomForestClassifier, BaggingClassifier
from sklearn.linear_model import SGDClassifier
import xgboost as xgb
from sklearn.neighbors import KNeighborsClassifier
import lightgbm as lgb

knn = KNeighborsClassifier(n_neighbors=2, p=1)
lr = LogisticRegression(C = 1 ,random_state = 42, max_iter = 150, solver = 'liblinear')
rf = RandomForestClassifier(class_weight='balanced', n_estimators = 100,max_features = 5, max_depth = 7, min_samples_leaf = 9, random_state = 42, n_jobs = -1)
sgd = SGDClassifier(alpha=0.01, max_iter=75, n_jobs=-1, random_state=21)
bc = BaggingClassifier(base_estimator=DecisionTreeClassifier(), max_features=6,
                  max_samples=60, n_estimators=50, random_state=1)
xgb_clf = xgb.XGBClassifier(base_score=0.5, objective='binary:logistic', n_estimators=300,seed=1, learning_rate=0.1, max_depth=10)
LGBM = lgb.LGBMClassifier(boosting_type = 'gbdt', learning_rate = 0.1, max_depth = 10, n_estimators = 200, num_leaves = 31, random_state = 42, n_jobs=- 1)

classifiers = [
              ('Logistic Regression', lr),
              ('Random Forest', rf),
              ('LGBMClassifier', LGBM),
              #('sgd', sgd),
               #('knn', knn),
               ('xgb', xgb_clf)]
              #('BaggingClassifier', bc)]  

for clf_name , clf in classifiers:
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_val)
    print('{:s} : {:.4f}'.format(clf_name, accuracy_score(y_val, y_pred)))
    print(f"f1 score: {f1_score(y_val, y_pred)}")
    # recall score
    print(f"recall_score: {recall_score(y_val, y_pred)}")
vc = VotingClassifier(estimators=classifiers)
vc.fit(X, y)

y_pred = vc.predict(X_val)
print('VotingClassifier: {:.4f}'.format(accuracy_score(y_val, y_pred)))
print(f"f1 score: {f1_score(y_val, y_pred)}")
# recall score
print(f"recall_score: {recall_score(y_val, y_pred)}")<jupyter_output>/opt/conda/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
  warnings.warn(label_encoder_deprecation_msg, UserWarning)<jupyter_text>Confusion MatrixConfusion matrix is a tool to evaluate the accuracy of a classification process. The horizontal axis shows the labels that the model predicted and the vertical axis shows the real labels. The number in each cell indicates the number of images that match the selected combination. The diagonal shows the number of images in which the model correctly predicted reality.<jupyter_code>import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay, classification_report
from sklearn.metrics import confusion_matrix

cm_plot_labels = [x for x in range(2)]

cm = confusion_matrix(y_val, y_pred)

disp = ConfusionMatrixDisplay(confusion_matrix=cm,
                              display_labels=cm_plot_labels)

disp = disp.plot(include_values=True,
                 cmap=plt.cm.Blues, ax=None, xticks_rotation='horizontal')

plt.show()
print(confusion_matrix(y_val, y_pred))<jupyter_output>[[4047  253]
 [ 503  945]]<jupyter_text>5. Results<jupyter_code>y_pred_test = vc.predict(test_df)

output = pd.DataFrame({'enrollee_id':test.enrollee_id.astype(np.int32), 'target':y_pred_test})
output.to_csv('my_submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13279751575486345, 'fraction_numerical': 0.03501080768411118, 'mean_word_length': 5.86478578892372, 'pattern_counts': {'":': 0, '<': 119, '<?xml version=': 0, '>': 120, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '304579', 'license': 'MIT', 'n_tokens_mistral': 12192, 'n_tokens_neox': 11158, 'n_words': 2386, 'path': 'job_change_predictor.ipynb', 'repo_name': 'nettasav/job_change_predictor', 'size': '32847'}