<jupyter_start><jupyter_text>Classification Metrics Classification problems are perhaps the most common type of machine learning problem and assuch there are a myriad of metrics that can be used to evaluate predictions for these problems.In this section we will review how to use the following metrics:1. Classification Accuracy.- Logarithmic Loss.- Area Under ROC Curve.- Confusion Matrix.- Classification Report.  Load Data<jupyter_code>#Load Data
from pandas import read_csv
from sklearn.linear_model import LogisticRegression
filename = 'pima-indians-diabetes.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(filename, names=names)
array = dataframe.values
X = array[:,0:8]
Y = array[:,8]<jupyter_output><empty_output><jupyter_text>Classification Accuracy<jupyter_code># Cross Validation Classification Accuracy
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
kfold = KFold(n_splits=10, random_state=7)
model = LogisticRegression()
scoring = 'accuracy'
results = cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
print("Accuracy: %.3f (%.3f)" % (results.mean(), results.std()))<jupyter_output>Accuracy: 0.770 (0.048)<jupyter_text>Logarithmic Loss<jupyter_code># Cross Validation Classification LogLoss
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
kfold = KFold(n_splits=10, random_state=7)
model = LogisticRegression()
scoring = 'neg_log_loss'
results = cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
print("Accuracy: %.3f (%.3f)" % (results.mean(), results.std()))<jupyter_output>c:\users\hasan\appdata\local\programs\python\python36-32\lib\site-packages\sklearn\linear_model\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
c:\users\hasan\appdata\local\programs\python\python36-32\lib\site-packages\sklearn\linear_model\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
c:\users\hasan\appdata\local\programs\python\python36-32\lib\site-packages\sklearn\linear_model\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
c:\users\hasan\appdata\local\programs\python\python36-32\lib\site-packages\sklearn\linear_model\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
c:\users\hasan\appdata\local\programs\python\pyt[...]<jupyter_text>Area Under ROC Curve<jupyter_code># Cross Validation Classification ROC AUC
# An area of 1.0 represents a model that made all predictions perfectly. An area of 0.5 represents a model that is as good as 
# random. 
scoring = 'roc_auc'
results = cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
print("AUC: %.3f (%.3f)" % (results.mean(), results.std()))<jupyter_output>AUC: 0.824 (0.041)<jupyter_text>Confusion Matrix<jupyter_code># Cross Validation Classification Confusion Matrix
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
test_size = 0.33
seed = 7
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=test_size,
random_state=seed)
model = LogisticRegression()
model.fit(X_train, Y_train)
predicted = model.predict(X_test)
matrix = confusion_matrix(Y_test, predicted)
print(matrix)<jupyter_output>[[141  21]
 [ 41  51]]<jupyter_text>Classification Report<jupyter_code># Cross Validation Classification Report
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
test_size = 0.33
seed = 7
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=test_size,
random_state=seed)
model = LogisticRegression()
model.fit(X_train, Y_train)
predicted = model.predict(X_test)
report = classification_report(Y_test, predicted)
print(report)<jupyter_output>precision    recall  f1-score   support

         0.0       0.77      0.87      0.82       162
         1.0       0.71      0.55      0.62        92

   micro avg       0.76      0.76      0.76       254
   macro avg       0.74      0.71      0.72       254
weighted avg       0.75      0.76      0.75       254<jupyter_text>Regression Metrics In this section will review 3 of the most common metrics for evaluating predictions on regressionmachine learning problems:1. Mean Absolute Error.- Mean Squared Error.- R2.<jupyter_code>from pandas import read_csv
from sklearn.linear_model import LinearRegression
filename = 'housing.csv'
names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO',
'B', 'LSTAT', 'MEDV']
dataframe = read_csv(filename, names=names)
array = dataframe.values
X = array[:,0:13]
Y = array[:,13]<jupyter_output><empty_output><jupyter_text>Mean Absolute Error<jupyter_code># Cross Validation Regression MAE
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
kfold = KFold(n_splits=10, random_state=7)
model = LinearRegression()
scoring = 'neg_mean_absolute_error'
results = cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
print("MAE: %.3f (%.3f)" % (results.mean(), results.std()))<jupyter_output>MAE: -4.005 (2.084)<jupyter_text>A value of 0 indicates no error or perfect predictions. Like logloss, this metric is inverted bythe cross val score() function.  Mean Squared Error<jupyter_code># Cross Validation Regression MSE
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
num_folds = 10
kfold = KFold(n_splits=10, random_state=7)
model = LinearRegression()
scoring = 'neg_mean_squared_error'
results = cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
print("MSE: %.3f (%.3f)" % (results.mean(), results.std()))<jupyter_output>MSE: -34.705 (45.574)<jupyter_text>R2 Metric<jupyter_code>kfold = KFold(n_splits=10, random_state=7)
model = LinearRegression()
scoring = 'r2'
results = cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
print("R^2: %.3f (%.3f)" % (results.mean(), results.std()))<jupyter_output>R^2: 0.203 (0.595)
{'fraction_non_alphanumeric': 0.11154690931910888, 'fraction_numerical': 0.033417006589268904, 'mean_word_length': 5.61993769470405, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '662309', 'license': 'MIT', 'n_tokens_mistral': 2287, 'n_tokens_neox': 2127, 'n_words': 562, 'path': 'Machine Learning Algorithm Performance Metrics.ipynb', 'repo_name': 'ahkhalwai/Machine_Learning_Mastery_With_Python_Book', 'size': '6374'}