<jupyter_start><jupyter_text>4. Evaluating a model Once you've trained a model, you'll want a way to measure how trustworthy its predictions are.  Scikit-Learn implements 3 different methods of evaluating models.1. Estimator score() method. Calling score() on a model instance will return a metric assosciated with the type of model you're using. The metric depends on which model you're using.2. Scoring parameter. This parameter can be passed to methods such as cross_val_score() or GridSearchCV() to tell Scikit-Learn to use a specific type of scoring metric.3. Problem-specific metric functions. Similar to how the scoring parameter can be passed different scoring functions, Scikit-Learn implements these as stand alone functions. The scoring function you use will also depend on the problem you're working on.Classification problems have different evaluation metrics and scoring functions to regression problems.  4.1 Evaluation a model with score() method - it is a quick evaluation classification problem example (building a classifier to predict whether or not someone has heart disease based on their medical records)<jupyter_code>%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
heart_df = pd.read_csv("data/heart-disease.csv")
heart_df.head() # classification dataset - supervised learning
# No. of samples in the dataset
len(heart_df)
from sklearn.model_selection import train_test_split

# Import the RandomForestClassifier model class from the ensemble module
from sklearn.ensemble import RandomForestClassifier

# Setup random seed
np.random.seed(42)

# Split the data into X (features/data) and y (target/labels)
X = heart_df.drop("target", axis=1)
y = heart_df["target"]

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Instantiate the model (on the training set)
clf = RandomForestClassifier()

# Call the fit method on the model and pass it training data
clf.fit(X_train, y_train);<jupyter_output><empty_output><jupyter_text>Once the model has been fit on the training data (X_train, y_train), we can call the score() method on it and evaluate our model on the test data, data the model has never seen before (X_test, y_test).<jupyter_code># Check the score of the model (on the test set)
heart_score = clf.score(X_test, y_test)
print(f"Score = {heart_score*100}%")<jupyter_output>Score = 85.24590163934425%<jupyter_text>clf is an instance of RandomForestClassifier, the score() method uses mean accuracy as its score method -- default metric for classification  score() makes predictions on X_test using the trained model and then compares those predictions to the actual labels y_test.  Remember, you can find this by pressing SHIFT + TAB within the brackets of score() when called on a model instance. regression model<jupyter_code># Import the Boston housing dataset of SKlearn - built in regression dataset
from sklearn.datasets import load_boston
boston = load_boston()
# Covert it to a pandas dataframe - for better inspection

# take the data key, and label the columns
boston_df = pd.DataFrame(boston["data"],columns=boston["feature_names"])

# create a target column in df by using target values from dataset
boston_df["target"] = pd.Series(boston["target"])
boston_df
# Import the RandomForestRegressor model class from the ensemble module
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

# Setup random seed
np.random.seed(42)

# Create the data
X = boston_df.drop("target", axis=1)
y = boston_df["target"]

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Institate and fit the model (on the training set)
model = RandomForestRegressor()
model.fit(X_train, y_train)
# Check the score of the model (on the test set)
reg_score = model.score(X_test, y_test)
print(f"Score = {reg_score*100}%")<jupyter_output>Score = 86.54448653350507%
{'fraction_non_alphanumeric': 0.06956303365143145, 'fraction_numerical': 0.013812154696132596, 'mean_word_length': 5.540229885057471, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '99555', 'license': 'MIT', 'n_tokens_mistral': 1190, 'n_tokens_neox': 1126, 'n_words': 493, 'path': 'Scikit-Learn/4) Evaluating a Model/EvaluationModel_example1_score.ipynb', 'repo_name': 'Jawwad-Fida/Jupyter-Notebooks', 'size': '3982'}