<jupyter_start><jupyter_text>Graduate Admissions  ContextThis dataset is created for prediction of Graduate Admissions from an Indian perspective. ContentThe dataset contains several parameters which are considered important during the application for Masters Programs. The parameters included are : 1. GRE Scores ( out of 340 ) 2. TOEFL Scores ( out of 120 ) 3. University Rating ( out of 5 ) 4. Statement of Purpose and Letter of Recommendation Strength ( out of 5 ) 5. Undergraduate GPA ( out of 10 ) 6. Research Experience ( either 0 or 1 ) 7. Chance of Admit ( ranging from 0 to 1 )<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
import warnings
warnings.filterwarnings('ignore')
import numpy as np

graduate=pd.read_csv('Admission_Predict.csv')
graduate.head()
graduate.info()
graduate.describe()
graduate.drop('Serial No.',axis=1, inplace=True)
graduate.head()
graduate.isnull().any()
graduate=graduate.rename(columns = {'Chance of Admit ':'Chance of Admit'})
fig,ax = plt.subplots(figsize=(10, 10))
sns.heatmap(graduate.corr(), ax=ax, annot=True, linewidths=0.05, fmt= '.2f',cmap="magma")
plt.show()

sns.pairplot(graduate)<jupyter_output><empty_output><jupyter_text>Linear Regression<jupyter_code>import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn import linear_model
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

sns.jointplot("CGPA", "Chance of Admit", data=graduate, kind="reg", color="m", height=7)
X_cgpa=graduate[["CGPA"]]
y=graduate[["Chance of Admit"]]

X_cgpa_train,X_cgpa_test,y_train,y_test = train_test_split(X_cgpa,y,test_size=.3,random_state=42)

regr = LinearRegression()
regr.fit(X_cgpa_train, y_train)
y_pred= regr.predict(X_cgpa_test)

# Plot outputs
plt.figure()
plt.plot(X_cgpa_test,y_pred, color='blue', linewidth=3)
sns.scatterplot(x="CGPA", y="Chance of Admit" ,data=graduate)

# The coefficients
print('Coefficients: \n', regr.coef_,regr.intercept_)
# Explained variance score: 1 is perfect prediction
print('r2_score: %.2f' % r2_score(y_test, y_pred))
plt.scatter(regr.predict(X_cgpa_test),regr.predict(X_cgpa_test)-y_pred,c='b',s=40,alpha=0.5)
plt.ylabel('Residuals')
sns.jointplot("GRE Score", "Chance of Admit", data=graduate, kind="reg", color="r", height=7)
X_gre=graduate[["GRE Score"]]
y=graduate[["Chance of Admit"]]

X_gre_train,X_gre_test,y_train,y_test = train_test_split(X_gre,y,test_size=.3,random_state=42)

regr = LinearRegression()
regr.fit(X_gre_train, y_train)
y_pred= regr.predict(X_gre_test)

# Plot outputs
plt.figure()
plt.plot(X_gre_test,y_pred, color='blue', linewidth=3)
sns.scatterplot(x="GRE Score", y="Chance of Admit" ,data=graduate)

# The coefficients
print('Coefficients: \n', regr.coef_,regr.intercept_)
# Explained variance score: 1 is perfect prediction
print('r2_score: %.2f' % r2_score(y_test, y_pred))
plt.scatter(regr.predict(X_gre_test),regr.predict(X_gre_test)-y_pred,c='b',s=40,alpha=0.5)
plt.ylabel('Residuals')
sns.jointplot("TOEFL Score", "Chance of Admit", data=graduate, kind="reg", color="g", height=7)
X_toefl=graduate[["TOEFL Score"]] 
y=graduate[["Chance of Admit"]]

X_toefl_train,X_toefl_test,y_train,y_test = train_test_split(X_toefl,y,test_size=.3,random_state=42)

regr = LinearRegression()
regr.fit(X_toefl_train, y_train)
y_pred= regr.predict(X_toefl_test)
# Plot outputs
plt.figure()
plt.plot(X_toefl_test,y_pred, color='blue', linewidth=3)
sns.scatterplot(x="TOEFL Score", y="Chance of Admit" ,data=graduate)
# The coefficients
print('Coefficients: \n', regr.coef_,regr.intercept_)
# Explained variance score: 1 is perfect prediction
print('r2_score: %.2f' % r2_score(y_test, y_pred))
plt.scatter(regr.predict(X_toefl_test),regr.predict(X_toefl_test)-y_pred,c='b',s=40,alpha=0.5)
plt.ylabel('Residuals')<jupyter_output><empty_output><jupyter_text>In the above we modelled considering each correlated separately and we have gained very useful insights. Now let us fit considering all feaures together    Regression  Linear Regression<jupyter_code>y = graduate["Chance of Admit"].values
x = graduate.drop(["Chance of Admit"],axis=1)

# separating train (80%) and test (%20) sets
from sklearn.model_selection import train_test_split

x_train, x_test,y_train, y_test = train_test_split(x,y,test_size = 0.20,random_state = 42)<jupyter_output><empty_output><jupyter_text>Normalization (min-max scaling):It makes values scaled to a fixed range (0-1).<jupyter_code>from sklearn.preprocessing import MinMaxScaler
scalerX = MinMaxScaler(feature_range=(0, 1))
x_train[x_train.columns] = scalerX.fit_transform(x_train[x_train.columns])
x_test[x_test.columns] = scalerX.transform(x_test[x_test.columns])

lr = LinearRegression()
lr.fit(x_train,y_train)
y_head_lr = lr.predict(x_test)

print("real value of y_test[1]: " + str(y_test[1]) + " -> the predict: " + str(lr.predict(x_test.iloc[[1],:])))
print("real value of y_test[2]: " + str(y_test[2]) + " -> the predict: " + str(lr.predict(x_test.iloc[[2],:])))

from sklearn.metrics import r2_score
print("r_2 score: ", r2_score(y_test,y_head_lr))

y_head_lr_train = lr.predict(x_train)
print("r_2 score (train dataset): ", r2_score(y_train,y_head_lr_train))<jupyter_output>real value of y_test[1]: 0.68 -> the predict: [0.72368741]
real value of y_test[2]: 0.9 -> the predict: [0.93536809]
r_2 score:  0.821208259148699
r_2 score (train dataset):  0.7951946003191086<jupyter_text>Random Forest regression<jupyter_code>from sklearn.ensemble import RandomForestRegressor
rfr = RandomForestRegressor(n_estimators = 100, random_state = 42)
rfr.fit(x_train,y_train)
y_head_rfr = rfr.predict(x_test) 

from sklearn.metrics import r2_score
print("r_2 score: ", r2_score(y_test,y_head_rfr))
print("real value of y_test[1]: " + str(y_test[1]) + " -> the predict: " + str(rfr.predict(x_test.iloc[[1],:])))
print("real value of y_test[2]: " + str(y_test[2]) + " -> the predict: " + str(rfr.predict(x_test.iloc[[2],:])))


y_head_rf_train = rfr.predict(x_train)
print("r_2 score (train dataset): ", r2_score(y_train,y_head_rf_train))<jupyter_output>r_2 score:  0.8074111823415694
real value of y_test[1]: 0.68 -> the predict: [0.7249]
real value of y_test[2]: 0.9 -> the predict: [0.9407]
r_2 score (train dataset):  0.9634880602889714<jupyter_text>Decision Tree Regression<jupyter_code>from sklearn.tree import DecisionTreeRegressor
dtr = DecisionTreeRegressor(random_state = 42)
dtr.fit(x_train,y_train)
y_head_dtr = dtr.predict(x_test) 

from sklearn.metrics import r2_score
print("r_2 score: ", r2_score(y_test,y_head_dtr))
print("real value of y_test[1]: " + str(y_test[1]) + " -> the predict: " + str(dtr.predict(x_test.iloc[[1],:])))
print("real value of y_test[2]: " + str(y_test[2]) + " -> the predict: " + str(dtr.predict(x_test.iloc[[2],:])))

y_head_dtr_train = dtr.predict(x_train)
print("r_2 score (train dataset): ", r2_score(y_train,y_head_dtr_train))<jupyter_output>r_2 score:  0.6262105228127393
real value of y_test[1]: 0.68 -> the predict: [0.73]
real value of y_test[2]: 0.9 -> the predict: [0.94]
r_2 score (train dataset):  1.0<jupyter_text>Bayesian regression<jupyter_code>from sklearn import linear_model

bayes = linear_model.BayesianRidge()
bayes.fit(x_train,y_train)
y_head_bayes = bayes.predict(x_test) 

from sklearn.metrics import r2_score
print("r_2 score: ", r2_score(y_test,y_head_bayes))
print("real value of y_test[1]: " + str(y_test[1]) + " -> the predict: " + str(bayes.predict(x_test.iloc[[1],:])))
print("real value of y_test[2]: " + str(y_test[2]) + " -> the predict: " + str(bayes.predict(x_test.iloc[[2],:])))

y_head_bayes_train = bayes.predict(x_train)
print("r_2 score (train dataset): ", r2_score(y_train,y_head_bayes_train))<jupyter_output>r_2 score:  0.819644590071969
real value of y_test[1]: 0.68 -> the predict: [0.72287936]
real value of y_test[2]: 0.9 -> the predict: [0.93366298]
r_2 score (train dataset):  0.7949612441808608<jupyter_text>Support Vector Machine (SVM)<jupyter_code>from sklearn import svm
vector = svm.SVR()
vector.fit(x_train,y_train)
y_head_vector = vector.predict(x_test) 

from sklearn.metrics import r2_score
print("r_2 score: ", r2_score(y_test,y_head_vector))
print("real value of y_test[1]: " + str(y_test[1]) + " -> the predict: " + str(vector.predict(x_test.iloc[[1],:])))
print("real value of y_test[2]: " + str(y_test[2]) + " -> the predict: " + str(vector.predict(x_test.iloc[[2],:])))

y_head_vector_train = vector.predict(x_train)
print("r_2 score (train dataset): ", r2_score(y_train,y_head_vector_train))<jupyter_output>r_2 score:  0.7533377635332518
real value of y_test[1]: 0.68 -> the predict: [0.69740123]
real value of y_test[2]: 0.9 -> the predict: [0.85627533]
r_2 score (train dataset):  0.7211185608124213<jupyter_text>Gaussian Process<jupyter_code>from sklearn.gaussian_process import GaussianProcessRegressor
gpr = GaussianProcessRegressor(random_state=0)
gpr.fit(x_train,y_train)
y_head_gpr = gpr.predict(x_test) 

from sklearn.metrics import r2_score
print("r_2 score: ", r2_score(y_test,y_head_gpr))
print("real value of y_test[1]: " + str(y_test[1]) + " -> the predict: " + str(gpr.predict(x_test.iloc[[1],:])))
print("real value of y_test[2]: " + str(y_test[2]) + " -> the predict: " + str(gpr.predict(x_test.iloc[[2],:])))

y_head_gpr_train = gpr.predict(x_train)
print("r_2 score (train dataset): ", r2_score(y_train,y_head_gpr_train))<jupyter_output>r_2 score:  -17.281231410482793
real value of y_test[1]: 0.68 -> the predict: [0.96785909]
real value of y_test[2]: 0.9 -> the predict: [1.11619506]
r_2 score (train dataset):  0.9999993885121039<jupyter_text>Comparing the regression models<jupyter_code>y = np.array([r2_score(y_test,y_head_lr),r2_score(y_test,y_head_rfr),r2_score(y_test,y_head_dtr),r2_score(y_test,y_head_bayes),r2_score(y_test,y_head_vector)])
x = ["LinearRegression","RandomForestReg.","DecisionTreeReg.",'BayesianReg','SVM']
plt.bar(x,y)
plt.title("Comparison of Regression Algorithms")
plt.xticks(rotation=45)
plt.xlabel("Regressor")
plt.ylabel("r2_score")
plt.show()<jupyter_output><empty_output><jupyter_text>Comments: Linear and Bayesian are the best regression models  Classification<jupyter_code>from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

graduate=pd.read_csv('Admission_Predict.csv')

# it may be needed in the future.
serialNo = graduate["Serial No."].values
graduate.drop(["Serial No."],axis=1,inplace = True)

graduate=graduate.rename(columns = {'Chance of Admit ':'Chance of Admit'})
y = graduate["Chance of Admit"].values
x = graduate.drop(["Chance of Admit"],axis=1)

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

X_train,X_test,y_train,y_test = train_test_split(x,y,test_size=.2,random_state=42)

scaler = MinMaxScaler()# escala as features entre 0 e 1.

X_train_scaled = scaler.fit_transform(X_train)
X_train = pd.DataFrame(X_train_scaled)

X_test_scaled = scaler.fit_transform(X_test)
X_test = pd.DataFrame(X_test_scaled)

y_train_01 = [1 if each > 0.8 else 0 for each in y_train]
y_test_01  = [1 if each > 0.8 else 0 for each in y_test]

# list to array
y_train_01 = np.array(y_train_01)
y_test_01 = np.array(y_test_01)
seed = 7
scoring = 'accuracy'

models = []
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))

# evaluate each model in turn
results = []
names = []
for name, model in models:
	kfold = model_selection.KFold(n_splits=10, random_state=seed)
	cv_results = model_selection.cross_val_score(model, X_train, y_train_01,cv=kfold, scoring=scoring)
	results.append(cv_results)
	names.append(name)
	msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
	print(msg)


from sklearn import metrics

lr=LogisticRegression().fit(X_train,y_train_01)
prob_lr=lr.predict_proba(X_train)

lda=LinearDiscriminantAnalysis().fit(X_train,y_train_01)
prob_lda=lda.predict_proba(X_train)

knn=KNeighborsClassifier().fit(X_train,y_train_01)
prob_knn=knn.predict_proba(X_train)

cart=DecisionTreeClassifier().fit(X_train,y_train_01)
prob_cart=cart.predict_proba(X_train)

gnb=GaussianNB().fit(X_train,y_train_01)
prob_gnb=gnb.predict_proba(X_train)

svm=SVC(probability=True).fit(X_train,y_train_01)
prob_svm=svm.predict_proba(X_train)


#Compute the ROC curve: true positives/false positives

tpr_lr,fpr_lr,thresh_lr=metrics.roc_curve(y_train_01,prob_lr[:,0])
tpr_lda,fpr_lda,thresh_lda=metrics.roc_curve(y_train_01,prob_lda[:,0])
tpr_knn,fpr_knn,thresh_knn=metrics.roc_curve(y_train_01,prob_knn[:,0])
tpr_cart,fpr_cart,thresh_cart=metrics.roc_curve(y_train_01,prob_cart[:,0])
tpr_gnb,fpr_gnb,thresh_gnb=metrics.roc_curve(y_train_01,prob_gnb[:,0])
tpr_svm,fpr_svm,thresh_svm=metrics.roc_curve(y_train_01,prob_svm[:,0])

#Area under Curve (AUC)
from sklearn.metrics import auc

roc_auc_lr = auc(fpr_lr, tpr_lr)
roc_auc_lda = auc(fpr_lda, tpr_lda)
roc_auc_knn = auc(fpr_knn, tpr_knn)
roc_auc_cart = auc(fpr_cart, tpr_cart)
roc_auc_gnb = auc(fpr_gnb, tpr_gnb)
roc_auc_svm = auc(fpr_svm, tpr_svm)

#Plotting the ROC curves


plt.figure()
plt.plot([0, 1], [0, 1], 'k--')
plt.plot(fpr_lr, tpr_lr, label='LR, ROC curve (area = %0.2f)' % roc_auc_lr)
plt.plot(fpr_lda, tpr_lda, label='LDA, ROC curve (area = %0.2f)' % roc_auc_lda)
plt.plot(fpr_knn, tpr_knn, label='KNN, ROC curve (area = %0.2f)' % roc_auc_knn)
plt.plot(fpr_cart, tpr_cart, label='CART, ROC curve (area = %0.2f)' % roc_auc_cart)
plt.plot(fpr_gnb, tpr_gnb, label='NB, ROC curve (area = %0.2f)' % roc_auc_gnb)
plt.plot(fpr_svm, tpr_svm, label='SVC, ROC curve (area = %0.2f)' % roc_auc_svm)
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.title('ROC curve')
plt.legend(loc='best')
plt.show()

# Make predictions on validation dataset
print("--------------------------")
print("LogisticRegression Report")
print("--------------------------")
predictions_lr = lr.predict(X_test)
print("accuracy =",accuracy_score(y_test_01, predictions_lr))
print("confusion matrix",confusion_matrix(y_test_01, predictions_lr))
print(classification_report(y_test_01, predictions_lr))

print("--------------------------")
print("LinearDiscriminantAnalysis Report")
print("--------------------------")
predictions_lda = lda.predict(X_test)
print("accuracy =",accuracy_score(y_test_01, predictions_lda))
print("confusion matrix",confusion_matrix(y_test_01, predictions_lda))
print(classification_report(y_test_01, predictions_lda))

print("--------------------------")
print("KNeighborsClassifier Report")
print("--------------------------")
predictions_knn = knn.predict(X_test)
print("accuracy =",accuracy_score(y_test_01, predictions_knn))
print("confusion matrix",confusion_matrix(y_test_01, predictions_knn))
print(classification_report(y_test_01, predictions_knn))

print("--------------------------")
print("DecisionTreeClassifier Report")
print("--------------------------")
predictions = cart.predict(X_test)
print("accuracy =",accuracy_score(y_test_01, predictions))
print("confusion matrix",confusion_matrix(y_test_01, predictions))
print(classification_report(y_test_01, predictions))

print("--------------------------")
print("GaussianNB Report")
print("--------------------------")
predictions_gnb = gnb.predict(X_test)
print("accuracy =",accuracy_score(y_test_01, predictions_gnb))
print("confusion matrix",confusion_matrix(y_test_01, predictions_gnb))
print(classification_report(y_test_01, predictions_gnb))

print("--------------------------")
print("SVC Report")
print("--------------------------")
predictions_svm = svm.predict(X_test)
print("accuracy =",accuracy_score(y_test_01, predictions_svm))
print("confusion matrix",confusion_matrix(y_test_01, predictions_svm))
print(classification_report(y_test_01, predictions_svm))<jupyter_output>LR: 0.918750 (0.061237)
LDA: 0.934375 (0.045178)
KNN: 0.900000 (0.058962)
CART: 0.862500 (0.067315)
NB: 0.890625 (0.064424)
SVM: 0.921875 (0.070226)
{'fraction_non_alphanumeric': 0.15761429003935815, 'fraction_numerical': 0.041961852861035424, 'mean_word_length': 6.6997668997668995, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '67732', 'license': 'MIT', 'n_tokens_mistral': 6829, 'n_tokens_neox': 6376, 'n_words': 1181, 'path': 'graduate_admissions.ipynb', 'repo_name': 'alefefreire/MLtraining', 'size': '16515'}