<jupyter_start><jupyter_text>Welcome to this Kernel *** This kernel is a compilation of tricks of sklearn published by <NAME> weekly.***You can find the original tricks and tips on the GitHub repo:https://github.com/justmarkham/scikit-learn-tips *** This kernel is under construction. I will be updating it regularly.***  Table of contents[Importing libraries and setting some helper functions](Imports)[Trick 35: Passing a df directly to sklearn](trick35)[Trick 34: Feature selection with Pipeline](trick34)[Trick 33: Using custom and existing function in a ColumnTransformer](trick33)[Trick 32: Area Under Curve (AUC) for binary classification: ovo and ovr strategies](trick32)[Trick 31: Shuffle when using cross_val_score](trick31)[Trick 30: Four ways of displaying the model coefficients](trick30)[Trick 29: Vectorize two text columns using ColumnTransformer](trick29)[Trick 28: Save a model of pipeline using joblib](trick28)[Trick 27: Imputing missing values for categorical values](trick27)[Trick 26: Use of stratify when performing classification problems](trick26)[Trick 25: Prunning decision trees (new in 0.22 and above).](trick25)[Trick 24: Plotting the decision tree with sklearn (new in 0.21 and above)](trick24)[Trick 23: Display the intercept & coefficients for a liner model](trick23)[Trick 22: Two types of Pipelines](trick22)[Trick 21: Several ROC curves in a single plot (new in sklearn 0.22)](trick21)[Trick 20: Plot confusion matrix (new in sklearn 0.22)](trick20)[Trick 19: Most important parameters of a LogisticRegression](trick19)[Trick 18: Convert your GridSearchCV or RandomizedGridSearch results into a pandas DataFrame](trick18)[Trick 17: RandomizedGridSearch](trick17)[Trick 16: Crossvalidate and gridsearch a sklearn pipeline](trick16)[Trick 15: OneHotEncoder: tips using it](trick15)[Trick 14: Handling missing values](trick14)[Trick 13: Examine each step of a Pipeline](trick13)[Trick 12: Difference between Pipeline and make_pipeline](trick12)[Trick 11: KNNImputer](trick11)[Trick 10: Using random_state to reproduce results](trick10)[Trick 9: Using missing values as a feature: SimpleImputer & add_indicator = True](trick9)[Trick 8: Using make_pipeline in a ML project](trick8)[Trick 7: Handle new data while using OneHotEncoder](trick7)[Trick 6: Common ways to encode categorical features: OneHotEncoder, OrdinalEncoder](trick6)[Trick 5: Benefits of using sklearn for preprocessing and not pandas](trick5)[Trick 4: When to use fit_transform and transform methods](trick4)[Trick 3: Difference between fit and transform method](trick3)[Trick 2: Seven ways to select columns using ColumnsTransformer](trick2)[Trick 1: Using column selector to transform different columns](trick1)  Importing libraries and setting some helper functions[Go back to the Table of Contents](table_of_contents)<jupyter_code># basic libraries
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from IPython.display import Image

# this will allow us to print all the files as we generate more in the kernel
def print_files(directory = "output"):
    if directory.lower() == "input":
        for dirname, _, filenames in os.walk('/kaggle/input'):
            for filename in filenames:
                print(os.path.join(dirname, filename))
    else:
        for dirname, _, filenames in os.walk('/kaggle/working'):
            for filename in filenames:
                print(os.path.join(dirname, filename))

# check Trick 91 for an example
def generate_sample_data(): # creates a fake df for testing
    number_or_rows = 20
    num_cols = 7
    cols = list("ABCDEFG")
    df = pd.DataFrame(np.random.randint(1, 20, size = (number_or_rows, num_cols)), columns=cols)
    df.index = pd.util.testing.makeIntIndex(number_or_rows)
    return df

# check Trick 91 for an example
def generate_sample_data_datetime(): # creates a fake df for testing
    number_or_rows = 365*24
    num_cols = 2
    cols = ["sales", "customers"]
    df = pd.DataFrame(np.random.randint(1, 20, size = (number_or_rows, num_cols)), columns=cols)
    df.index = pd.util.testing.makeDateIndex(number_or_rows, freq="H")
    return df

# show several prints in one cell. This will allow us to condence every trick in one cell.
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

print_files("input")<jupyter_output>/kaggle/input/logisticregression/LogisticRegression.jpg
/kaggle/input/titanic/train.csv
/kaggle/input/titanic/gender_submission.csv
/kaggle/input/titanic/test.csv
/kaggle/input/roc-curve/ROC Curve.jpeg<jupyter_text>Trick 35: Passing a df directly to sklearn[Go back to the Table of Contents](table_of_contents)<jupyter_code># We don't need to use .values when passing a df or a pandas series to sklearn
# It knows internally how to acess the values and deal with them

#------------------------------------------------------------
# import libraries
from sklearn.linear_model import LogisticRegression

#------------------------------------------------------------
# get the data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
X = df[['Pclass', 'Fare']]
y = df["Survived"]

#------------------------------------------------------------
# check the X and y types
print(type(X))
print(type(y))

#------------------------------------------------------------
# instanciate our classes
model = LogisticRegression()

# we fit directly a df and a series and sklearn deals with the rest
model.fit(X, y)<jupyter_output><class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.series.Series'><jupyter_text>Trick 34: Feature selection with Pipeline[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import SelectPercentile, chi2

#------------------------------------------------------------
# get the data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
X = df["Name"]
y = df["Survived"]

#------------------------------------------------------------
# instanciate our classes
vectorizer = CountVectorizer()
model = LogisticRegression()

#------------------------------------------------------------
# make the pipeline without feature selection
pipe = make_pipeline(vectorizer, model)
score = cross_val_score(pipe, X, y, scoring = 'accuracy').mean()
print("Score of pipeline without feature selection is {}".format(score))

#------------------------------------------------------------
# make the pipeline without feature selection

# keep 50% of features with the best chi-squared scores
selection = SelectPercentile(chi2, percentile = 50)

# add the selection after preprocessing but before model
pipe = make_pipeline(vectorizer, selection, model)
score = cross_val_score(pipe, X, y, scoring = 'accuracy').mean()
print("Score of pipeline with feature selection is {}".format(score))<jupyter_output>Score of pipeline without feature selection is 0.7957190383528967
Score of pipeline with feature selection is 0.8147824995292197<jupyter_text>Trick 33: Using custom and existing function in a ColumnTransformer[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import FunctionTransformer

#------------------------------------------------------------
# get some data
X = pd.DataFrame({'Fare':[200, 300, 50, 900],
                  'Code':['X12', 'Y20', 'Z7', np.nan],
                  'Deck':['A101', 'C102', 'A200', 'C300']})

#------------------------------------------------------------
# use an existing column and make the function compatible with Pipeline
clip_values = FunctionTransformer(np.clip, kw_args={'a_min':100, 'a_max':600})

#------------------------------------------------------------
# create a custom function
def first_letter(string_column):
    return string_column.apply(lambda x: x.str.slice(0, 1))

# now use FunctionTransformer to make the function compatible with Pipeline
get_first_letter = FunctionTransformer(first_letter)

#------------------------------------------------------------
# create the column Transformer
ct = make_column_transformer(
    (clip_values, ['Fare']),
    (get_first_letter, ['Code', 'Deck']))

#------------------------------------------------------------
# Original X
print("Original X")
X

#------------------------------------------------------------
# Modified X
print("Modified X")
ct.fit_transform(X)<jupyter_output>Original X<jupyter_text>Trick 32: Area Under Curve (AUC) for binary classification: ovo and ovr strategies[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_score

# get X and y for classification
X, y = load_wine(return_X_y=True)

# select only a few features
X = X[:, 0:2]

# instanciate the model for regression
model_clf = LogisticRegression()

# Multiclass AUC with train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
model_clf.fit(X_train, y_train)
y_score = model_clf.predict_proba(X_test)

# use 'ovo' (One-vs-One) or 'ovr' (One-vs-Rest)
print("roc_auc_score is {} for one vs one strategy with train test split".format(roc_auc_score(y_test, y_score, multi_class = 'ovo')))
print("roc_auc_score is {} for one vs rest strategy with train test split".format(roc_auc_score(y_test, y_score, multi_class = 'ovr')))
print("-----------------------------------------")

# Multiclass AUC with cross-validation
# use 'roc_auc_ovo' (One-vs-One) or 'roc_auc_ovr' (One-vs-Rest)
print("cross_val_score is {} for one vs one strategy with cross validation".format(cross_val_score(model_clf, X, y, cv = 5, scoring = 'roc_auc_ovo').mean()))
print("cross_val_score is {} for one vs rest strategy with cross validation".format(cross_val_score(model_clf, X, y, cv = 5, scoring = 'roc_auc_ovr').mean()));<jupyter_output>roc_auc_score is 0.9399801587301587 for one vs one strategy with train test split
roc_auc_score is 0.9484053018535777 for one vs rest strategy with train test split
-----------------------------------------
cross_val_score is 0.9086960878627546 for one vs one strategy with cross validation
cross_val_score is 0.9123529143963927 for one vs rest strategy with cross validation<jupyter_text>Trick 31: Shuffle when using cross_val_score[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold, StratifiedKFold

# get X and y for regression
X_reg, y_reg = load_diabetes(return_X_y = True)

# get X and y for classification
df = pd.read_csv("/kaggle/input/titanic/train.csv", usecols = ['Pclass', 'Fare', 'SibSp', 'Survived']).dropna()

# separate X and y
X_clf = df[['Pclass', 'Fare', 'SibSp']]
y_clf = df[["Survived"]]

# instanciate the model for regression
model_reg = LinearRegression()

# instanciate the model for classification
model_clf = LogisticRegression()

# Use KFold for regression
kf = KFold(5, shuffle = True, random_state = 1)
print("cross_val_score for regression model")
cross_val_score(model_reg, # the regression model, in our case, LinearRegression
                X_reg, # X: features to learn from
                y_reg, # y: what the predict
                cv = kf, # cross_validation scheme we have created earlier
                scoring = "r2") # metric to use to validate the quality of the model

# Use StratifiedKFold for classification
skf = StratifiedKFold(5, shuffle = True, random_state = 1)
print("cross_val_score for classification model")
cross_val_score(model_clf, # the model, in our case, LogisticRegression
                X_clf, # X: features to learn from
                y_clf, # y: what the predict
                cv = skf, # cross_validation scheme we have created earlier
                scoring = "accuracy") # metric to use to validate the quality of the model<jupyter_output>cross_val_score for regression model<jupyter_text>Trick 30: Four ways of displaying the model coefficients[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline

#------------------------------------------------------------
# instanciate the classes
ohe = OneHotEncoder()
model = LogisticRegression()

#------------------------------------------------------------
# get the data
df = pd.read_csv("/kaggle/input/titanic/train.csv", usecols = ['Embarked', 'Survived']).dropna()

# separate X and y
X = df[["Embarked"]]
y = df[["Survived"]]

# create a pipeline and fit the X and y
pipe = Pipeline([("ohe", ohe),
                 ("clf", model)])
pipe.fit(X, y)

# inspect the coefficients
print("1 way to show model coefficients")
pipe.named_steps.clf.coef_
print("2 way to show model coefficients")
pipe.named_steps["clf"].coef_
print("3 way to show model coefficients")
pipe["clf"].coef_
print("4 way to show model coefficients")
pipe[1].coef_<jupyter_output>/opt/conda/lib/python3.6/site-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)<jupyter_text>Trick 29: Vectorize two text columns using ColumnTransformer[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.compose import make_column_transformer

#------------------------------------------------------------
# get train data
df_train = pd.read_csv("/kaggle/input/titanic/train.csv")

# drop any nans
X = df_train[["Name", "Cabin"]].dropna()

#------------------------------------------------------------
# instanciate CountVectorizer
count_vect = CountVectorizer()

#------------------------------------------------------------
# instanciate CountVectorizer
# You can pass the CountVectorizer multiple times and it will learn
# separate vocabularies.
# to do so, you must use make_column_transformer
ct = make_column_transformer((count_vect, 'Name'), (count_vect, 'Cabin'))
X_transform = ct.fit_transform(X)
X_transform<jupyter_output><empty_output><jupyter_text>Trick 28: Save a model of pipeline using joblib[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
import joblib

#------------------------------------------------------------
# get train data
df_train = pd.read_csv("/kaggle/input/titanic/train.csv")

# drop any nans
df_train.dropna(axis = "rows", inplace = True)

# separate X and y
cols_for_x = ["Embarked", "Sex"]
X_train = df_train[cols_for_x]
y_train = df_train["Survived"]

#------------------------------------------------------------
# get test data
df_test = pd.read_csv("/kaggle/input/titanic/test.csv")

# drop any nans
df_test.dropna(axis = "rows", inplace = True)

X_test = df_test[cols_for_x]

#------------------------------------------------------------
# instanciate Ohe and make_pipeline
ohe = OneHotEncoder()
model = LogisticRegression()

#------------------------------------------------------------
# create the pipeline
pipe = make_pipeline(ohe, model)

#------------------------------------------------------------
# predict_using pipeline
pipe.fit(X_train, y_train)

#------------------------------------------------------------
# save pipeline
joblib.dump(pipe, 'pipe.joblib')

# print our newly saved pipeline
print_files()

#------------------------------------------------------------
# save pipeline
new_pipe = joblib.load('/kaggle/working/pipe.joblib')

#------------------------------------------------------------
# predict using the same pipe and the old pipe
print("------------")
print("Old pipe.")
pipe.predict(X_test)
print("------------")
print("Old pipe.")
new_pipe.predict(X_test)
print("Notice that both pipes predict the same result.")<jupyter_output><empty_output><jupyter_text>Trick 27: Imputing missing values for categorical values[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# get some fake data
d = {"Shape_Original":["square", "square", "square", "oval", "circle", np.nan]}
df = pd.DataFrame(d)

#------------------------------------------------------------
# import libraries
from sklearn.impute import SimpleImputer

#------------------------------------------------------------
# impute values using most frequent
df["most_frequent"] = SimpleImputer(strategy = "most_frequent").fit_transform(df[["Shape_Original"]])

#------------------------------------------------------------
# impute values using most constant
df["constant"]  = SimpleImputer(strategy = "constant", fill_value = "missing").fit_transform(df[["Shape_Original"]])

#------------------------------------------------------------
# the result of our imputation
df.style.apply(lambda x: ['background: lightgreen' if x.name == 5 else '' for i in x], axis=1)<jupyter_output><empty_output><jupyter_text>Trick 26: Use of stratify when performing classification problems[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.model_selection import train_test_split

#------------------------------------------------------------
# generate some data and separate X and y
df = pd.DataFrame({'feature':list(range(8)), 'target':['not fraud']*6 + ['fraud']*2})
X = df[['feature']]
y = df['target']

#------------------------------------------------------------
# train and test without stratify

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.5, random_state = 1)

print("y_train withous stratify")
y_train
print("y_test withous stratify")
y_test


#------------------------------------------------------------
# train and test with stratify

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.5, stratify = y, random_state = 1)

print("Notice how using statify preserves the fraud and not fraud percentage.")
print("y_train with stratify")
y_train
print("y_test with stratify")
y_test<jupyter_output>y_train withous stratify<jupyter_text>Trick 25: Prunning decision trees (new in 0.22 and above).[Go back to the Table of Contents](table_of_contents)<jupyter_code>import sklearn
print(sklearn.__version__)

#------------------------------------------------------------
# import libraries
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
df['Sex'] = df['Sex'].map({'male':0, 'female':1})
X = df[["Pclass", "Fare", "Sex"]]
y = df["Survived"]

#------------------------------------------------------------
# basic model and evaluation
model = DecisionTreeClassifier(random_state = 175)
model.fit(X, y)
score = cross_val_score(model, X, y, scoring = "accuracy")
print("Our DecissionTree with {} nodes has scored {}".format(model.tree_.node_count, score.mean()))

#------------------------------------------------------------
# prun the tree and see cross validation score
# Notice that the score went up. Prunnig trees has a lot of benefits, the main one is reducing overfitting.
# ccp_alpha is the parameter that controls the decision tree complexity (cost complexity parameter).
# Greater values of ccp_alpha increase the number of nodes pruned.
# documentation 
# https://scikit-learn.org/stable/auto_examples/tree/plot_cost_complexity_pruning.html

model = DecisionTreeClassifier(ccp_alpha = 0.001, random_state = 175)
model.fit(X, y)
score = cross_val_score(model, X, y, scoring = "accuracy")
print("Our prunned DecissionTree with {} nodes has scored {}".format(model.tree_.node_count, score.mean()))<jupyter_output>0.22.2.post1<jupyter_text>Trick 24: Plotting the decision tree with sklearn (new in 0.21 and above).[Go back to the Table of Contents](table_of_contents)<jupyter_code>import sklearn
print(sklearn.__version__)

#------------------------------------------------------------
# import libraries
from sklearn.model_selection import train_test_split
from sklearn.metrics import plot_confusion_matrix
from sklearn.tree import DecisionTreeClassifier, plot_tree, export_text

#------------------------------------------------------------
# create our instances
model = DecisionTreeClassifier()

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
df['Sex'] = df['Sex'].map({'male':0, 'female':1})
X = df[["Pclass", "Fare", "Sex"]]
y = df["Survived"]

features = ["Pclass", "Fare", "Sex"]
classes = ["Survived"]

#------------------------------------------------------------
# train test split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

#------------------------------------------------------------
# fit and predict

model.fit(X_train, y_train)

#------------------------------------------------------------
# plot the tree

plt.figure(figsize = (20, 10))
plot_tree(model, feature_names = features, filled = True);
#------------------------------------------------------------
# show the text
# I will plot only the first 200 characters of the tree since it grows rapidly
print(export_text(model, feature_names = features, show_weights=True)[:200]);<jupyter_output>|--- Sex <= 0.50
|   |--- Fare <= 26.27
|   |   |--- Fare <= 7.91
|   |   |   |--- Fare <= 7.86
|   |   |   |   |--- Fare <= 7.79
|   |   |   |   |   |--- Pclass <= 2.50
|   |   |   |   |   |   |--- w<jupyter_text>Trick 23: Display the intercept & coefficients for a liner model[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression

#------------------------------------------------------------
# load data and separate X and y
dataset = load_diabetes()
X, y = dataset.data, dataset.target
features = dataset.feature_names

#------------------------------------------------------------
# fit model
model = LinearRegression()
model.fit(X, y)

#------------------------------------------------------------
# intercept and coef
model.intercept_
model.coef_
list(zip(features, model.coef_))<jupyter_output><empty_output><jupyter_text>Trick 22: Two types of Pipelines[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# Two types of ROC Curve

# If the pipeline ends in a classifier or regressor, you use the fit and predict methods
# If the pipeline ends in a transformer you use the fit_transform and transform methods

path = "/kaggle/input/roc-curve/ROC Curve.jpeg"
Image(path)<jupyter_output><empty_output><jupyter_text>Trick 21: Several ROC curves in a single plot (new in sklearn 0.22)[Go back to the Table of Contents](table_of_contents)<jupyter_code>import sklearn
print(sklearn.__version__)

#------------------------------------------------------------
# import libraries
from sklearn.model_selection import train_test_split
from sklearn.metrics import plot_roc_curve
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier

#------------------------------------------------------------
# create our instances
lr = LogisticRegression()
dt = DecisionTreeClassifier()
rf = RandomForestClassifier()

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
X = df[["Pclass", "Fare"]]
y = df["Survived"]

#------------------------------------------------------------
# train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

#------------------------------------------------------------
# fit and predict
lr.fit(X_train, y_train)
dt.fit(X_train, y_train)
rf.fit(X_train, y_train)

#------------------------------------------------------------
# plot roc curve
disp = plot_roc_curve(lr, X_test, y_test)
plot_roc_curve(dt, X_test, y_test, ax = disp.ax_) # ax = disp.ax_ this line will share the x axis
plot_roc_curve(rf, X_test, y_test, ax = disp.ax_)<jupyter_output>0.22.2.post1<jupyter_text>Trick 20: Plot confusion matrix (new in sklearn 0.22)[Go back to the Table of Contents](table_of_contents)<jupyter_code>import sklearn
print(sklearn.__version__)

#------------------------------------------------------------
# import libraries
from sklearn.model_selection import train_test_split
from sklearn.metrics import plot_confusion_matrix
from sklearn.linear_model import LogisticRegression

#------------------------------------------------------------
# create our instances
model = LogisticRegression(random_state = 1)

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
X = df[["Pclass", "Fare"]]
y = df["Survived"]

#------------------------------------------------------------
# train test split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

#------------------------------------------------------------
# fit and predict

model.fit(X_train, y_train)

#------------------------------------------------------------
# plot confusion matrix
# notice that you have to pass the model, X_test and y_test
# plot_confusion_matrix predicts with the model and plots the values

disp = plot_confusion_matrix(model, X_test, y_test, cmap = "Blues", values_format = ".3g")

print("The classical confusion matrix")
disp.confusion_matrix<jupyter_output>0.22.2.post1<jupyter_text>Trick 19: Most important parameters of a LogisticRegression[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# C: inverse of regularization strength
# penalty: type of regularization
# solver: algorithm used for optimization

path = "/kaggle/input/logisticregression/LogisticRegression.jpg"
Image(path)<jupyter_output><empty_output><jupyter_text>Trick 18: Convert your GridSearchCV or RandomizedGridSearch results into a pandas DataFrame[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.preprocessing import OneHotEncoder
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.linear_model import LogisticRegression

#------------------------------------------------------------
# create our instances
ohe = OneHotEncoder()
vect = CountVectorizer()
ct = make_column_transformer((ohe, ["Sex"]), (vect, "Name"))
model = LogisticRegression(solver = "liblinear", random_state = 1)

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
X = df[["Sex", "Name", "Fare"]]
y = df["Survived"]

#------------------------------------------------------------
# make pipeline
pipeline = make_pipeline(ct, model)

#------------------------------------------------------------
# cross validate the entire pipeline
print("Notice the score of our entire pipeline is {}".format(cross_val_score(pipeline, X, y, cv = 5, scoring = "accuracy").mean()))
cross_val_score(pipeline, X, y, cv = 5, scoring = "accuracy").mean()

#------------------------------------------------------------
# gridsearch the entire pipeline

# set the parameters
params = {"columntransformer__countvectorizer__min_df":[1, 2],
         "logisticregression__C":[0.1, 1, 10],
         "logisticregression__penalty":["l1", "l2"]}

grid = GridSearchCV(pipeline, params, cv = 5, scoring = "accuracy")
grid.fit(X, y)

# convert to a pandas DataFrame

results = pd.DataFrame(grid.cv_results_)[["params", "mean_test_score", "rank_test_score"]]
results.sort_values("rank_test_score")<jupyter_output>Notice the score of our entire pipeline is 0.8024543343167408<jupyter_text>Trick 17: RandomizedGridSearch[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import RandomizedSearchCV, cross_val_score
from sklearn.naive_bayes import MultinomialNB
import scipy as sp

#------------------------------------------------------------
# create our instances
vect = CountVectorizer()
model = MultinomialNB()

#------------------------------------------------------------
# make pipeline
pipeline = make_pipeline(vect, model)

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
X = df['Name']
y = df['Survived']

#------------------------------------------------------------
# set the params to optimize

params = {}

params["countvectorizer__min_df"] = [1, 2, 3, 4]
params["countvectorizer__lowercase"] = [True, False]
params["multinomialnb__alpha"] = sp.stats.uniform(scale = 1)

#------------------------------------------------------------
# optimize

rand = RandomizedSearchCV(pipeline, params, n_iter = 10, cv = 5, scoring = "accuracy", random_state = 1)
rand.fit(X, y)

#------------------------------------------------------------
# best score and params

print("Best score achieved with our search is:")
rand.best_score_

print("Best params are:")
rand.best_params_<jupyter_output><empty_output><jupyter_text>Trick 16: Crossvalidate and gridsearch a sklearn pipeline[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.preprocessing import OneHotEncoder
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.linear_model import LogisticRegression

#------------------------------------------------------------
# create our instances
ohe = OneHotEncoder()
vect = CountVectorizer()
ct = make_column_transformer((ohe, ["Sex"]), (vect, "Name"))
model = LogisticRegression(solver = "liblinear", random_state = 1)

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
X = df[["Sex", "Name", "Fare"]]
y = df["Survived"]

#------------------------------------------------------------
# make pipeline
pipeline = make_pipeline(ct, model)

#------------------------------------------------------------
# cross validate the entire pipeline
print("Notice the score of our entire pipeline is {}".format(cross_val_score(pipeline, X, y, cv = 5, scoring = "accuracy").mean()))
cross_val_score(pipeline, X, y, cv = 5, scoring = "accuracy").mean()

#------------------------------------------------------------
# gridsearch the entire pipeline

# set the parameters
params = {"columntransformer__countvectorizer__min_df":[1, 2],
         "logisticregression__C":[0.1, 1, 10],
         "logisticregression__penalty":["l1", "l2"]}

grid = GridSearchCV(pipeline, params, cv = 5, scoring = "accuracy")
grid.fit(X, y)

# see the best score
print("#-------------------------------------------------------------------------")
print("Best score of the GridSearchCV is ")
grid.best_score_

# see the best params
print("#-------------------------------------------------------------------------")
print("Best parameters of the GridSearchCV are ")
grid.best_params_<jupyter_output>Notice the score of our entire pipeline is 0.8024543343167408<jupyter_text>Trick 15: OneHotEncoder: tips using it[Go back to the Table of Contents](table_of_contents) If you use OneHotEncoder, don't "drop = 'first'":1. Multicollinearity is rarely and issue with sklearn models2. drop = 'first' is incompatible with handle_unknown = 'ignore'3. May cause you problems if yoy standarize all features or use a regularized model  Trick 14: Handling missing values[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# If you have missing values you can:
# 1. Drop all rows with missing values
# 2. Drop all colmns with missing values
# 3. Impute missing values
# 4. Use a model that handles missing values

#------------------------------------------------------------
# import libraries
from sklearn.experimental import enable_hist_gradient_boosting # this import enables "experimental" packages and clases in sklearn
from sklearn.ensemble import HistGradientBoostingClassifier # this is an experimental package
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
print("We can see that we have missing values")

df.isnull().sum()

#------------------------------------------------------------
# split target and feature

features = [col for col in df.columns if df[col].dtype != "object"] # select only numerical columns
features.remove("Survived")
features.remove("PassengerId")

X = df[features]
y = df["Survived"]

#------------------------------------------------------------
# Train a model that handles missing values

model = HistGradientBoostingClassifier()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1, random_state = 175)

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print("Accuracy of our model while having missing values is {}%".format(round(accuracy_score(y_test, y_pred), 2)*100))<jupyter_output>We can see that we have missing values<jupyter_text>Trick 13: Examine each step of a Pipeline[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

#------------------------------------------------------------
# create each instance
si = SimpleImputer()
model = LogisticRegression()

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")

#------------------------------------------------------------
# select columns to transform
X = df[["Fare", "Age"]].head()
X["Age"].iloc[0] = np.nan # create a missing values
X.head()

y = df[["Survived"]].head()

#------------------------------------------------------------
# use make_pipeline

pipeline = make_pipeline(si, model)

pipeline.fit(X, y)

#------------------------------------------------------------
# let's see the statistics of each step
print("These are the imputed values with the SimpleImputer")
pipeline.named_steps.simpleimputer.statistics_

print("Display the coefficients of the linear model")
pipeline.named_steps.logisticregression.coef_<jupyter_output><empty_output><jupyter_text>Trick 12: Difference between Pipeline and make_pipeline[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.compose import make_column_transformer, ColumnTransformer
from sklearn.pipeline import make_pipeline, Pipeline

#------------------------------------------------------------
# create each instance
ohe = OneHotEncoder()
si = SimpleImputer()
model = LogisticRegression()

#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")

#------------------------------------------------------------
# select columns to transform
X = df[["Fare", "Embarked", "Sex", "Age"]].head()
X["Age"].iloc[0] = np.nan # create a missing values
X.head()

y = df[["Survived"]].head()

#------------------------------------------------------------
# use make_pipeline

column_transformer = make_column_transformer(
(ohe, ["Embarked", "Sex"]),
(si, ["Age"]),
remainder = "passthrough"
)

pipeline = make_pipeline(column_transformer, model)

pipeline.fit(X, y)

#------------------------------------------------------------
# use Pipeline
# The main difference is that we must name each step

column_transformer = ColumnTransformer(
[("encoder", ohe, ["Embarked", "Sex"]), # notice how we must name each step
("imputer", si, ["Age"])],
remainder = "passthrough"
)

pipeline = Pipeline([("preprocessing", column_transformer), ("model", model)]) # notice how we must name each step
pipeline.fit(X, y)<jupyter_output><empty_output><jupyter_text>Trick 11: KNNImputer[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.impute import KNNImputer

#------------------------------------------------------------
# create some random data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
df.isnull().sum() # inspect nulls
df.head()

#------------------------------------------------------------
# let's see KNNInputer in action
knn_inputer = KNNImputer()

X = df[["SibSp", "Fare", "Age"]]
nan_index = X[X["Age"].isnull()].index

print("Data with nans")
X[X.index.isin(nan_index)].head(10)

print("Transformed data with no nans, and the values are based on the NN.")
X_transformed = pd.DataFrame(knn_inputer.fit_transform(X), columns = ["SibSp", "Fare", "Age"], index = X.index)
X_transformed[X.index.isin(nan_index)].head(10)<jupyter_output><empty_output><jupyter_text>Trick 10: Using random_state to reproduce results[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import libraries
from sklearn.model_selection import train_test_split

#------------------------------------------------------------
# create some random data

df = generate_sample_data()
target = [np.random.choice([0, 1]) for i in range(len(df))]
df["target"] = target
print("A sneak peak at our df.")
df.head(3)

#------------------------------------------------------------
# train and target columns
features = list(df.columns)[:-1] # all except the last one

#------------------------------------------------------------
# split nr 1 using random_state to reproduce results
X_train1, X_test1, y_train1, y_test1 = train_test_split(df[features], df["target"], test_size = 0.1, random_state = 175)

#------------------------------------------------------------
# split nr 2 using random_state to reproduce results
X_train2, X_test2, y_train2, y_test2 = train_test_split(df[features], df["target"], test_size = 0.1, random_state = 175)

#------------------------------------------------------------
# split nr 3 using with no random_state
X_train3, X_test3, y_train3, y_test3 = train_test_split(df[features], df["target"], test_size = 0.1)

#------------------------------------------------------------
# let's look at our results
print("First train test split.")
X_train1.head()
print("Second train test split. Equal to the first one.")
X_train2.head()
X_train1.index == X_train2.index
print("Third train test split. Different than the others.")
X_train3.head()
X_train1.index == X_train3.index<jupyter_output>A sneak peak at our df.<jupyter_text>Trick 9: Using missing values as a feature: SimpleImputer & add_indicator = True[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import the libraries
from sklearn.impute import SimpleImputer

#------------------------------------------------------------
# create some train and test data

# create a train df
d = {
"Age":[10, 20, np.nan, 30, 15, 10, 40, 10, np.nan]
}

print("Train data")
df_train = pd.DataFrame(d)
df_train

#------------------------------------------------------------
# Sometimes there is a relathionship between missing values and the target
# We can use this information creating a new features while performing an imputation of a missing values

# normal SimpleImputer()
imputer = SimpleImputer()
df_transformed = imputer.fit_transform(df_train)
df_transformed

# SimpleImputer() with the parameter add_indicator
print("Notice aditional column with an indicator of 1 next to the previously missing values")
imputer = SimpleImputer(add_indicator = True)
df_transformed = imputer.fit_transform(df_train)
df_transformed<jupyter_output>Train data<jupyter_text>Trick 8: Using make_pipeline in a ML project[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import the libraries
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

#------------------------------------------------------------
# create some train and test data

# create a train df
d = {
"feat1":[10, 20, np.nan, 2],
"feat2":[25, 20, 5, 3],
"target":["A", "A", "B", "B"]
}

print("Train data")
df_train = pd.DataFrame(d)
df_train

# create a test df
d = {
"feat1":[30, 5, 15],
"feat2":[12, 10, np.nan]
}

print("Test data")
df_test = pd.DataFrame(d)
df_test

#------------------------------------------------------------
# simple ML project step by step

# create each instance
si = SimpleImputer()
model = LogisticRegression()
pipeline = make_pipeline(si, model)

# separate the data between target and features
features = ["feat1", "feat2"]
X_train, y_train = df_train[features], df_train["target"]
X_test = df_test[features]

#------------------------------------------------------------
# use pipeline to fit and predict

# First pipeline will use the SimpleImputer to imputer the missing values
# Then it will train using LogisticRegression
pipeline.fit(X_train, y_train)

# When used pipeline to predict, it will do the same steps as in fit
pipeline.predict(X_test)<jupyter_output>Train data<jupyter_text>Trick 7: Handle new data while using OneHotEncoder[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# create a train df
d = {
"Categorical":["A", "A", "B", "C"]
}

df_train = pd.DataFrame(d)
df_train

#------------------------------------------------------------
# import the libraries
from sklearn.preprocessing import OneHotEncoder

#------------------------------------------------------------
# transform data during train part
print("fit_transform using OneHotEncoder.")
ohe = OneHotEncoder(sparse = False, handle_unknown = "ignore") # if you don't put false, you will get a sparse matrix object
X_train = ohe.fit_transform(df_train[["Categorical"]])
X_train

#------------------------------------------------------------
# create a test df

d = {
"Categorical":["A", "A", "B", "C", "D"] # new value, D, previously not seen in train
}

df_test = pd.DataFrame(d)
df_test


print("transform using OneHotEncoder. Notice that we have a line with zeros for categorical value of D")
X_test = ohe.transform(df_test[["Categorical"]])
X_test<jupyter_output><empty_output><jupyter_text>Trick 6: Common ways to encode categorical features: OneHotEncoder, OrdinalEncoder[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# create a df
d = {
"Shape":["square", "square", "oval", "circle"],
"Class":["third", "first", "second", "third"],
"Size":["S", "S", "L", "XL"]
}

df = pd.DataFrame(d)
df

#------------------------------------------------------------
# import the libraries
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import OrdinalEncoder

#------------------------------------------------------------
# transform data using OneHotEncoder
print("Transform categorical data using OneHotEncoder")
ohe = OneHotEncoder(sparse = False) # if you don't put false, you will get a sparse matrix object
shaped_transformed = ohe.fit_transform(df[["Shape"]]) # if you pass as a series, you will need to reshape the data. Notice the double square bracket
shaped_transformed

#------------------------------------------------------------
# transform data using OrdinalEncoder
print("Transform categorical data using OrdinalEncoder")
print("When using OrdinalEncoder, your data has to have a order: like first class, second class, third class")
oe = OrdinalEncoder(categories = [["first", "second", "third"], # order for the column Class
                                  ["S", "M", "L", "XL"]]) # order for the column Size
categorical_ordinal_transformed = oe.fit_transform(df[["Class", "Size"]])
categorical_ordinal_transformed<jupyter_output><empty_output><jupyter_text>Trick 5: Benefits of using sklearn for preprocessing and not pandas[Go back to the Table of Contents](table_of_contents)  Reasons to use sklearn to ML preprocessing and not pandas1. You can cross-validate the entire workflow2. You can grid search model & preprocessing hyperparameters3. Avoids adding new columns to the source DataFrame4. pandas lacks separate fit/transform steps to prevent data leakage  Trick 4: When to use fit_transform and transform methods[Go back to the Table of Contents](table_of_contents)  ***Use "fit_transform" on training data, but "transform" (only) on testing/new data.***Applies the same transformations to both sets of data, which creates consistent columns and prevents data leakage!  Trick 3: Difference between fit and transform method[Go back to the Table of Contents](table_of_contents)  Q: What is the difference between the "fit" and "transform" methods? ***"fit"***: transformer learns something about the data ***"transform":*** it uses what it learned to do the data transformation------***CountVectorizer: ***fit: learns the vocabularytransform: creates a document term matrix using the vocabulary------***SimpleImputer: ***fit: learns the value to imputetransform: fills the missing value with the value to impute------***StandartScaler: ***fit: learns the mean and scale of each featuretransform: standarizes the features using the mean and scale------***HashingVectorizer: ***fit: if not used, then it's known as "stateless" transformertransform: creates a document term matrix using a hash of the token  Trick 2: Seven ways to select columns using ColumnsTransformer[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")

#------------------------------------------------------------
# select columns to transform
X = df[["Fare", "Embarked", "Sex", "Age"]].head()
X["Age"].iloc[0] = np.nan # create a missing values
X

#------------------------------------------------------------
# import the libraries
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import make_column_selector
from sklearn.compose import make_column_transformer

#------------------------------------------------------------
# instanciate the classes
ohe = OneHotEncoder()

#------------------------------------------------------------
# create the pipeline and select the columns by name

print("Select the column by name")

ct = make_column_transformer(
(ohe, ["Embarked", "Sex"])# if you have null values it will give and error. You must first fill those values before doing an ohe
)

# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed

#------------------------------------------------------------
# create the pipeline and select the columns by position

print("Select the column by position")

ct = make_column_transformer(
(ohe, [1, 2])# if you have null values it will give and error. You must first fill those values before doing an ohe
)

# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed

#------------------------------------------------------------
# create the pipeline and select the columns using slice

print("Select the column using slice")

ct = make_column_transformer(
(ohe, slice(1, 3))# if you have null values it will give and error. You must first fill those values before doing an ohe
)

# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed

#------------------------------------------------------------
# create the pipeline and select the columns using a boolean mask

print("Select the column using a boolean mask")

ct = make_column_transformer(
(ohe, [False, True, True, False])# if you have null values it will give and error. You must first fill those values before doing an ohe
)

# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed

#------------------------------------------------------------
# create the pipeline and select the columns using make_column_selector and regex

print("Select the column using using make_column_selector and regex. New in pandas 0.22")

ct = make_column_transformer(
(ohe, make_column_selector(pattern = "E|S"))# if you have null values it will give and error. You must first fill those values before doing an ohe
)

# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed

#------------------------------------------------------------
# create the pipeline and select the columns using make_column_selector and dtype_include

print("Select the column using using make_column_selector and dtype_include. New in pandas 0.22")

ct = make_column_transformer(
(ohe, make_column_selector(dtype_include = object))# if you have null values it will give and error. You must first fill those values before doing an ohe
)

# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed

#------------------------------------------------------------
# create the pipeline and select the columns using make_column_selector and dtype_exclude

print("Select the column using using make_column_selector and dtype_exclude. New in pandas 0.22")

ct = make_column_transformer(
(ohe, make_column_selector(dtype_exclude = "number"))# if you have null values it will give and error. You must first fill those values before doing an ohe
)

# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed<jupyter_output><empty_output><jupyter_text>Trick 1: Using ColumnTransformer to manipulate different columns[Go back to the Table of Contents](table_of_contents)<jupyter_code>#------------------------------------------------------------
# import data
df = pd.read_csv("/kaggle/input/titanic/train.csv")
df.head()

#------------------------------------------------------------
# select columns to transform
X = df[["Fare", "Embarked", "Sex", "Age"]].head()
X["Age"].iloc[0] = np.nan # create a missing values
X

#------------------------------------------------------------
# import the libraries
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import make_column_transformer

#------------------------------------------------------------
# instanciate the classes
ohe = OneHotEncoder()
imp = SimpleImputer()

#------------------------------------------------------------
# create the pipeline
ct = make_column_transformer(
(ohe, ["Embarked", "Sex"]), # if you have null values it will give and error. You must first fill those values before doing an ohe
(imp, ["Age"]), 
remainder = "passthrough" # this means that the column Fare will appear the last one.
)

#------------------------------------------------------------
# fit_transform the columns
X_transformed = ct.fit_transform(X) 
X_transformed<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.25994205952067423, 'fraction_numerical': 0.012321757778697469, 'mean_word_length': 2.344385026737968, 'pattern_counts': {'":': 21, '<': 120, '<?xml version=': 0, '>': 114, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '328624', 'license': 'Apache-2.0', 'n_tokens_mistral': 14902, 'n_tokens_neox': 14173, 'n_words': 4493, 'path': 'sklearn-35-best-tips-and-tricks.ipynb', 'repo_name': 'yqstar/Sklearn-Top35-Tricks', 'size': '53165'}