<jupyter_start><jupyter_text>Copyright 2020 <NAME> and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code.  Crossvalidation and Nested crossvalidationThe goal of all models we build is to **predict** and **generalize**.If our model only works for the data we train it on, and doesn't work for anything else, it's not a very useful model.One way to test generalization is to split the data into training and testing data.The training data is used to estimate the model's parameters, and the testing data is used to determine how much we can trust the model's predictions on unseen data.If the performance of the model is very good on the training data but very poor on the testing data, we say that the model has **overfit** the training data.If the model's performance on the testing data is as good or better than the training data, then we conclude that the model will generalize well.But what if we're wrong?What if we got lucky with a train/test split, such that the testing data is "easy" and very similar to the training data?One way to increase our confidence is to train and test **repeatedly with different train/test splits**.But again, we have a problem: what if our train/test splits overlap substantially?We only get more information about model performance if the splits are different, not when they are the same. What you will learnIn this notebook you will learn about methods to avoid overfit during training and to increase our confidence in model generalization. We will study the following:- Crossvalidation- Nested crossvalidation When to use crossvalidationIt's generally recommended to use crossvalidation in standard practice, rather than a single train/test split.Fortunately crossvalidation is so common that many data science frameworks make it relatively easy to use.An important consideration is what evaluation metrics mean when models are trained with crossvalidation, so keep evaluation metrics in mind as you progress through this notebook.  Crossvalidation**Crossvalidation** is a fairly simple yet elegant idea that solves these problems.Crossvalidation lets us use **all** our data for both training and testing, by partitioning it in separate sets, typically called **folds**. Figure 1 shows all the data split into 5 folds.Crossvalidation training would create five models using these five folds, such that each model uses a different fold for testing (blue).All the other folds are used for training.For example, one model would use fold 1 for testing and folds 2-5 for training. Figure 1. Crossvalidation with five folds. Source: Adapted from scikit-learn Using crossvalidation, all data is used for both training and testing **but not at the same time.**This allows us to train with all the data without worrying about overfitting. It also lets us robustly test generalization because we've used all the data for testing as well.Crossvalidation works nicely with all of the standard peformance metrics like $r^2$, accuracy, and precision/recall: because the predictions from separate models are on separate folds, we can calculate performance metrics as though the predictions came from a single model.Alternatively, we can calculate performance on each fold separately to get a distribution of performance.Finally, you might see the connections between crossvalidation and out-of-bag (OOB) error with bagging.Crossvalidation could be viewed as a generalization of OOB, since crossvalidation can be used with any model.Crossvalidation is also a simplification over OOB, since OOB requires keeping track of which aggregated models are "allowed" to make a prediction on a datapoint, whereas in crossvalidation, a single model makes predictions on its test fold, and that's it. Example: CrossvalidationLet's take a look at the `iris` dataset, which consists of measurements of sepals and petals, as well as the class label for three species of iris.| Variable                 | Type    | Description                                                                    |                                ||:--------------------------|:---------|:--------------------------------------------------------------------------------|--------------------------------|| Sepal Length             | Ratio   | Length of the leaves that support the petals                                    |                                || Sepal Width              | Ratio   | Width of the leaves that suppotr the petals                                     |                                || Petal Length             | Ratio   | Length of petals                                                               |                                || Petal Width              | Ratio   | Width of petals                                                                |                                || Species                  | Nominal     | Setosa, Versicolour, Virginica |      Source: This dataset was taken from the UCI Machine Learning Repository library     Because this is a familiar dataset, and because the focus is on crossvalidation, we will skip data exploration steps and go quickly to modeling.  Load dataImport `pandas` so we can load a dataframe:- `import pandas as pd`<jupyter_code>import pandas as pd

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="A]5Vf/x,=8dk=KKk3OrJ">pd</variable></variables><block type="importAs" id="G*NhDz5Jo?CcaJf3rUv}" x="150" y="308"><field name="libraryName">pandas</field><field name="libraryAlias" id="A]5Vf/x,=8dk=KKk3OrJ">pd</field></block></xml><jupyter_output><empty_output><jupyter_text>Load the dataframe using a CSV file:- Create `dataframe` and set it to `with pd do read_csv using` a list containing    - `"datasets/iris.csv"`- `dataframe`<jupyter_code>dataframe = pd.read_csv('datasets/iris.csv')

dataframe

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="Y+grzOaS*0=OHV6zO8m/">dataframe</variable><variable id="A]5Vf/x,=8dk=KKk3OrJ">pd</variable></variables><block type="variables_set" id="FJGldw(qFp?V=E]uaN)4" x="31" y="222"><field name="VAR" id="Y+grzOaS*0=OHV6zO8m/">dataframe</field><value name="VALUE"><block type="varDoMethod" id="4s18v%0iqE%_l@{fLYCo"><field name="VAR" id="A]5Vf/x,=8dk=KKk3OrJ">pd</field><field name="MEMBER">read_csv</field><data>pd:read_csv</data><value name="INPUT"><block type="lists_create_with" id="x*dHwc!e,/XY5e1C_ak1"><mutation items="1"></mutation><value name="ADD0"><block type="text" id="tN4Jmaffy/={ZFo;uu;W"><field name="TEXT">datasets/iris.csv</field></block></value></block></value></block></value></block><block type="variables_get" id="yN6MT1B[)Pk)cEECN_YZ" x="31" y="286"><field name="VAR" id="Y+grzOaS*0=OHV6zO8m/">dataframe</field></block></xml><jupyter_output><empty_output><jupyter_text>Prepare train/test sets Let's separate our predictors (`X`) from our class label (`Y`), putting each into its own dataframe:- Create `X` and set to `with dataframe do drop using` a list containing    - freestyle `columns=["Species"]`- Create `Y` and set to `dataframe [ ]` containing a list with `"Species"` inside<jupyter_code>X = dataframe.drop(columns=["Species"])
Y = dataframe[['Species']]

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="_z/H/tTnOYS1Lla2~;*0">X</variable><variable id="Y+grzOaS*0=OHV6zO8m/">dataframe</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="variables_set" id="M9g^9:hbZQ0Cb9jKq9V6" x="27" y="188"><field name="VAR" id="_z/H/tTnOYS1Lla2~;*0">X</field><value name="VALUE"><block type="varDoMethod" id="k7t38%eJz)@=B8{vW^(u"><field name="VAR" id="Y+grzOaS*0=OHV6zO8m/">dataframe</field><field name="MEMBER">drop</field><data>dataframe:drop</data><value name="INPUT"><block type="lists_create_with" id="GF+zQWW-#ABEFw$9!#^8"><mutation items="1"></mutation><value name="ADD0"><block type="dummyOutputCodeBlock" id="fk{P(H^@/_!^1+tw-Hi*"><field name="CODE">columns=["Species"]</field></block></value></block></value></block></value><next><block type="variables_set" id=":eCC|=CT/c:_fzcI.h5%"><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field><value name="VALUE"><block type="indexer" id="A%ev_@Jrd7:a(Gozw{Zv"><field name="VAR" id="Y+grzOaS*0=OHV6zO8m/">dataframe</field><value name="INDEX"><block type="lists_create_with" id="v`ZwcnYa3]GMVDC!T3al"><mutation items="1"></mutation><value name="ADD0"><block type="text" id="H9b;qzWmM({49_lwSk(2"><field name="TEXT">Species</field></block></value></block></value></block></value></block></next></block></xml><jupyter_output><empty_output><jupyter_text>Train model with crossvalidationWe need libraries for Gaussian naive Bayes, crossvalidation, and `ravel`- `import sklearn.model_selection as model_selection`- `import sklearn.naive_bayes as naive_bayes`- `import numpy as np`<jupyter_code>import sklearn.model_selection as model_selection
import sklearn.naive_bayes as naive_bayes
import numpy as np

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="-q~R*yx.VGQ@%!0Q+!gh">model_selection</variable><variable id="KA0k4+i.iu,J]5!9Nj%^">naive_bayes</variable><variable id="hxqH|@PJU~wrI]t*tHgd">np</variable></variables><block type="importAs" id="E{QJ$O@lc8u.;OTWdVgf" x="-8" y="173"><field name="libraryName">sklearn.model_selection</field><field name="libraryAlias" id="-q~R*yx.VGQ@%!0Q+!gh">model_selection</field><next><block type="importAs" id="s:sP+mk)a0E**6%-{LVV"><field name="libraryName">sklearn.naive_bayes</field><field name="libraryAlias" id="KA0k4+i.iu,J]5!9Nj%^">naive_bayes</field><next><block type="importAs" id="%54AMyx`5atYhM+,;y=E"><field name="libraryName">numpy</field><field name="libraryAlias" id="hxqH|@PJU~wrI]t*tHgd">np</field></block></next></block></next></block></xml><jupyter_output><empty_output><jupyter_text>There are several different options for crossvalidation output.The most straightforward is probably `cross_val_predict`, which takes a model, data, and specifications for crossvalidation, and then trains the model and makes predictions on the test folds:- Create variable `predictions`- Set it to `with model_selection do cross_val_predict using` a list containing    - `with naive_bayes create GaussianNB using`    - `X`    - `with np do ravel using` a list containing `Y`    - freestyle `cv=10` (for 10 folds)    **Note:** we can also use a pipeline instead of a vanilla model if we need to scale variables, etc.<jupyter_code>predictions = model_selection.cross_val_predict(naive_bayes.GaussianNB(), X, np.ravel(Y), cv=10)

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="(`;mrW|63Vww]$wlV9+1">predictions</variable><variable id="-q~R*yx.VGQ@%!0Q+!gh">model_selection</variable><variable id="KA0k4+i.iu,J]5!9Nj%^">naive_bayes</variable><variable id="_z/H/tTnOYS1Lla2~;*0">X</variable><variable id="hxqH|@PJU~wrI]t*tHgd">np</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="variables_set" id="v~v-8p`I+-]yk;np(M(P" x="-83" y="329"><field name="VAR" id="(`;mrW|63Vww]$wlV9+1">predictions</field><value name="VALUE"><block type="varDoMethod" id="8k.#5;5zB21(W~JU=mM["><field name="VAR" id="-q~R*yx.VGQ@%!0Q+!gh">model_selection</field><field name="MEMBER">cross_val_predict</field><data>model_selection:cross_val_predict</data><value name="INPUT"><block type="lists_create_with" id="JIEkdIPZPgt)o,O`zCMX"><mutation items="4"></mutation><value name="ADD0"><block type="varCreateObject" id="$gVob$hp3F-xl{I+UbzZ"><field name="VAR" id="KA0k4+i.iu,J]5!9Nj%^">naive_bayes</field><field name="MEMBER">GaussianNB</field><data>naive_bayes:GaussianNB</data></block></value><value name="ADD1"><block type="variables_get" id="+Q~gufbZ65hxF*8)q#$N"><field name="VAR" id="_z/H/tTnOYS1Lla2~;*0">X</field></block></value><value name="ADD2"><block type="varDoMethod" id="scikt5[Gq5cuXNTzF=]k"><field name="VAR" id="hxqH|@PJU~wrI]t*tHgd">np</field><field name="MEMBER">ravel</field><data>np:ravel</data><value name="INPUT"><block type="lists_create_with" id="=Mr6X5_/I`ansR9VIxbf"><mutation items="1"></mutation><value name="ADD0"><block type="variables_get" id="}VKVKM8x5o/F1Gu~)4Q$"><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field></block></value></block></value></block></value><value name="ADD3"><block type="dummyOutputCodeBlock" id="py$KuG=7@vxJRP2HGoIk"><field name="CODE">cv=10</field></block></value></block></value></block></value></block></xml><jupyter_output><empty_output><jupyter_text>Notice how much simpler this is than creating train/test splits!  Evaluate the model To measure performance, we need `sklearn.metrics`:- `import sklearn.metrics as metrics`<jupyter_code>import sklearn.metrics as metrics

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="jpz]0=,hLYv~rN^#0dFO">metrics</variable></variables><block type="importAs" id="{H02K*?VA0K3yr,FD/;h" x="51" y="95"><field name="libraryName">sklearn.metrics</field><field name="libraryAlias" id="jpz]0=,hLYv~rN^#0dFO">metrics</field></block></xml><jupyter_output><empty_output><jupyter_text>We can get the accuracy by comparing the predictions to *all* of `Y`:- `with metrics do accuracy_score using` a list containing    - `Y`    - `predictions`<jupyter_code>print(metrics.accuracy_score(Y, predictions))

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="jpz]0=,hLYv~rN^#0dFO">metrics</variable><variable id="(`;mrW|63Vww]$wlV9+1">predictions</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="text_print" id="u8M]eo-vRBhX!L/a@4+J" x="-102" y="208"><value name="TEXT"><shadow type="text" id=")gVsrXo+bq|?9m0cIU/Y"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="0B;?~qgT4?Xe8I|O=P)%"><field name="VAR" id="jpz]0=,hLYv~rN^#0dFO">metrics</field><field name="MEMBER">accuracy_score</field><data>metrics:</data><value name="INPUT"><block type="lists_create_with" id="qe7YbU#WU/F|Iuks0C:c"><mutation items="2"></mutation><value name="ADD0"><block type="variables_get" id="m^Z6y-8@L:s!oS)`%iJW"><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field></block></value><value name="ADD1"><block type="variables_get" id="|3h,TPit9wg+a;bR[r$P"><field name="VAR" id="(`;mrW|63Vww]$wlV9+1">predictions</field></block></value></block></value></block></value></block></xml><jupyter_output>0.9533333333333334<jupyter_text>And similarly we can get the recall and precision using all of `Y`:- `print with metrics do classification_report using` a list containing    - `Y`    - `predictions`<jupyter_code>print(metrics.classification_report(Y, predictions))

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="jpz]0=,hLYv~rN^#0dFO">metrics</variable><variable id="(`;mrW|63Vww]$wlV9+1">predictions</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="text_print" id="jFo8knaDn;JRZ=Fj7Tzo" x="57" y="316"><value name="TEXT"><shadow type="text" id="QWuxwm12CzEZ-r2sdwCR"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="(y`#h=AgNH6X+ahL$2L$"><field name="VAR" id="jpz]0=,hLYv~rN^#0dFO">metrics</field><field name="MEMBER">classification_report</field><data>metrics:classification_report</data><value name="INPUT"><block type="lists_create_with" id="A/#8KhfiGd[(@p,uaFFl"><mutation items="2"></mutation><value name="ADD0"><block type="variables_get" id="fQYj?Tz[y/t|%9~{2Pf["><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field></block></value><value name="ADD1"><block type="variables_get" id="mX(20KX!C)d_,/ddeysr"><field name="VAR" id="(`;mrW|63Vww]$wlV9+1">predictions</field></block></value></block></value></block></value></block></xml><jupyter_output>precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        50
  versicolor       0.92      0.94      0.93        50
   virginica       0.94      0.92      0.93        50

    accuracy                           0.95       150
   macro avg       0.95      0.95      0.95       150
weighted avg       0.95      0.95      0.95       150<jupyter_text>Because we split the data 10 different ways and trained 10 different models, we can be reasonably confident that we will see similar performance on new data.There are [other crossvalidation methods](https://scikit-learn.org/stable/modules/cross_validation.htmlcross-validation) besides `cross_val_predict` that return metrics per fold if you would like to see the range of performance across folds.  Nested crossvalidationNot only can crossvalidation help us better estimate model generalization, but it can also help us find **hyperparameters** of our models.A hyperparameter is something you specify when you create the model, as opposed to a parameter the model learns.Common examples we've encountered are the value for `K` in KNN, regularization parameters in ridge regression and lasso regression, and the `C` parameter for margin softness in SVM.Crossvalidation works for hyperparameters because the problem we face with them is much the same as with model fit: we can choose a hyperparameter that works well for some data, but it might not work well on other data.So if we use crossvalidation, we can get a good idea of how well a particular value for a hyperparameter works across the dataset.However, for hyperparameter crossvalidation to be really powerful, we need to to explore multiple candidate values for the hyperparameter, and then use crossvalidation for each one.**Grid search** is a simple way of defining candidate values for hyperparameters.Simply stated, vanilla grid search takes a list of candidate hyperparameter values you define and creates a separate model to evaluate each of those hyperparameter values.Grid search and crossvalidation work together like this:- Grid search gives crossvalidation a hyperparameter value- Crossvalidation builds as many models using that hyperparameter as there are folds- The average performance across folds is used to score the hyperparameter- Grid search gives crossvalidation another hyperparameter value and the process repeats- Once all hyperparameter values have been scored, the best is returned by grid searchWhile grid search and crossvalidation are very powerful together, they potentially create another problem for us.As we've discussed, we don't want to train and test with the same data, because then we don't know for sure our model will generalize.The problem is that if we do grid search to find hyperparameters on the same data we test our model on, then we've trained and tested with the same data.We could solve this problem by spliting our data into two sets, one for hyperparameter search + model training, and one for testing.However, that approach takes us back to the train/test split problem - what if we get a lucky split?We already solved the train/test split problem with crossvalidation, so can we solve this problem using crossvalidation too?The answer is yes, we can solve train/test splits for hyperparameters and normal model training at the same time using **nested crossvalidation**.Simply stated, nested crossvalidation partitions the data into folds and then partitions each of those folds into folds.The first set of folds (the **outer folds**) are used to train the model in the way we discussed in the last section.The second set of folds are used to evaluate the hyperparameters, e.g. using grid search.The reason that nested crossvalidation works is that when the model is tested on a fold, it hasn't used that fold to estimate its hyperparameters or its parameters.The process for one fold is shown in the figure below.Notice that the test component of this fold is not used to set hyperparameters nor is it used to train the model. Figure 2. Nested crossvalidation. Source: StackExchange  Example: Nested crossvalidationLet's continue our example with the `iris` dataset, but switch models to `SVC` with has the hyperparameter `C`.  Train model with nested crossvalidationWe need to import libraries for:- SVM- Scale (SVM is very sensitive to standardization)- Pipeline (to combine scaling and modeling)So we need the following imports:- `import sklearn.svm as svm`- `import sklearn.preprocessing as pp`- `import sklearn.pipeline as pipe`<jupyter_code>import sklearn.svm as svm
import sklearn.preprocessing as pp
import sklearn.pipeline as pipe

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="-1?@LYu7gyhX}3NV$-Lv">svm</variable><variable id=".T`xqZHAu7r0$MQ4s=Yg">pp</variable><variable id="=z9KfSBWpoDS,oj-?yx,">pipe</variable></variables><block type="importAs" id="tb8M1lM!u45Sg7*|pjG)" x="58" y="81"><field name="libraryName">sklearn.svm</field><field name="libraryAlias" id="-1?@LYu7gyhX}3NV$-Lv">svm</field><next><block type="importAs" id="*}7RY7RRNo7VC5@m%Tk["><field name="libraryName">sklearn.preprocessing</field><field name="libraryAlias" id=".T`xqZHAu7r0$MQ4s=Yg">pp</field><next><block type="importAs" id="+dJM+p.?OUVo4DOrpHa*"><field name="libraryName">sklearn.pipeline</field><field name="libraryAlias" id="=z9KfSBWpoDS,oj-?yx,">pipe</field></block></next></block></next></block></xml><jupyter_output><empty_output><jupyter_text>We're going to make a pipeline so we can scale and train in one step.However, we need to **name the stages** in order for grid search to work:- Create variable `model`- Set it to `with pipe create Pipeline using` a list containing a list containing    - a tuple (from LISTS; see picture below) containing        - `"scale"`        - `with pp create StandardScaler using`    - a tuple containing        - `"svm"`        - `with svm create SVC using` a list containing            - freestyle `random_state=1`            - freestyle `kernel="rbf"` (for a radial basis function kernel) ![image.png](attachment:image.png)<jupyter_code>model = pipe.Pipeline([('scale',(pp.StandardScaler())), ('svm',(svm.SVC(random_state=1, kernel="rbf")))])

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="WMn%x,]:2AW7#OLRLzEH">model</variable><variable id="=z9KfSBWpoDS,oj-?yx,">pipe</variable><variable id=".T`xqZHAu7r0$MQ4s=Yg">pp</variable><variable id="-1?@LYu7gyhX}3NV$-Lv">svm</variable></variables><block type="variables_set" id="}81D/tZY#o}$E:M}:u4x" x="185" y="368"><field name="VAR" id="WMn%x,]:2AW7#OLRLzEH">model</field><value name="VALUE"><block type="varCreateObject" id="p2Ty-;AM|3vXJ2XRy(7w"><field name="VAR" id="=z9KfSBWpoDS,oj-?yx,">pipe</field><field name="MEMBER">Pipeline</field><data>pipe:Pipeline</data><value name="INPUT"><block type="lists_create_with" id="q*C!dJG6s~6B9RsIRV(m"><mutation items="1"></mutation><value name="ADD0"><block type="lists_create_with" id="(Q-}ch^M{g#1!2LN+l,@"><mutation items="2"></mutation><value name="ADD0"><block type="tupleBlock" id="I_MY:8Z:j8lE@|Y8G)gY"><value name="FIRST"><block type="text" id="3om9JLcdxp[Hr$b~r9QQ"><field name="TEXT">scale</field></block></value><value name="SECOND"><block type="varCreateObject" id="e9G_EPu`q{*0`$!~:jh~"><field name="VAR" id=".T`xqZHAu7r0$MQ4s=Yg">pp</field><field name="MEMBER">StandardScaler</field><data>pp:StandardScaler</data></block></value></block></value><value name="ADD1"><block type="tupleBlock" id="sIhAnH/S~^@CGD:D*g=W"><value name="FIRST"><block type="text" id="^bl!01:~OK7|jQ^n[EW1"><field name="TEXT">svm</field></block></value><value name="SECOND"><block type="varCreateObject" id="ar7keIh-Yv)+b+#Edsp_"><field name="VAR" id="-1?@LYu7gyhX}3NV$-Lv">svm</field><field name="MEMBER">SVC</field><data>svm:SVC</data><value name="INPUT"><block type="lists_create_with" id="Al*7M]:C3p{sbqSe6!!_"><mutation items="2"></mutation><value name="ADD0"><block type="dummyOutputCodeBlock" id="Rijq=#hj2/,;|GT}ucWi"><field name="CODE">random_state=1</field></block></value><value name="ADD1"><block type="dummyOutputCodeBlock" id="^p.wPLUwvc9aiAUu=B4-"><field name="CODE">kernel="rbf"</field></block></value></block></value></block></value></block></value></block></value></block></value></block></value></block></xml><jupyter_output><empty_output><jupyter_text>This is our base model that we need to give to grid search, along with a parameter values we want grid search to try for `C`.Let's try the list `[1, 10, 100]`!- Create variable `gridSearch`- Set it to `with model_selection create GridSearchCV using` a list containing    - freestyle `estimator=` followed by `model`    - freestyle `param_grid={'svm__C': [1, 10, 100]}`    - freestyle `cv=10` (10 inner folds for grid search)- Set `predictions` to `with model_selection do cross_val_predict using` a list containing    - `gridSearch`    - `X`    - `with np do ravel using` a list containing `Y`    - freestyle `cv=10` (10 outer folds for model parameters)<jupyter_code>gridSearch = model_selection.GridSearchCV(estimator= model, param_grid={'svm__C': [1, 10, 100]}, cv=10)
predictions = model_selection.cross_val_predict(gridSearch, X, np.ravel(Y), cv=10)

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="yv+gPAZd/8:NDOnmJ~uf">gridSearch</variable><variable id="-q~R*yx.VGQ@%!0Q+!gh">model_selection</variable><variable id="(`;mrW|63Vww]$wlV9+1">predictions</variable><variable id="WMn%x,]:2AW7#OLRLzEH">model</variable><variable id="_z/H/tTnOYS1Lla2~;*0">X</variable><variable id="hxqH|@PJU~wrI]t*tHgd">np</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="variables_set" id="oYJd7r6+q~Z`__.-8mTl" x="-61" y="234"><field name="VAR" id="yv+gPAZd/8:NDOnmJ~uf">gridSearch</field><value name="VALUE"><block type="varCreateObject" id="jr4MyN[uTaUiK#OEw}*K"><field name="VAR" id="-q~R*yx.VGQ@%!0Q+!gh">model_selection</field><field name="MEMBER">GridSearchCV</field><data>model_selection:GridSearchCV</data><value name="INPUT"><block type="lists_create_with" id="?j)A.9BPO9u0XE:`(dAx"><mutation items="3"></mutation><value name="ADD0"><block type="valueOutputCodeBlock" id=";kaw*4PD92YnH+hAkZji"><field name="CODE">estimator=</field><value name="INPUT"><block type="variables_get" id="4JQDg_Q/4Ahisby=X0QT"><field name="VAR" id="WMn%x,]:2AW7#OLRLzEH">model</field></block></value></block></value><value name="ADD1"><block type="dummyOutputCodeBlock" id="LW0Qh];lc+lYES-:f}[l"><field name="CODE">param_grid={'svm__C': [1, 10, 100]}</field></block></value><value name="ADD2"><block type="dummyOutputCodeBlock" id="/qt+U,;Pr+V!yPkT@]?j"><field name="CODE">cv=10</field></block></value></block></value></block></value><next><block type="variables_set" id="v~v-8p`I+-]yk;np(M(P"><field name="VAR" id="(`;mrW|63Vww]$wlV9+1">predictions</field><value name="VALUE"><block type="varDoMethod" id="8k.#5;5zB21(W~JU=mM["><field name="VAR" id="-q~R*yx.VGQ@%!0Q+!gh">model_selection</field><field name="MEMBER">cross_val_predict</field><data>model_selection:cross_val_predict</data><value name="INPUT"><block type="lists_create_with" id="JIEkdIPZPgt)o,O`zCMX"><mutation items="4"></mutation><value name="ADD0"><block type="variables_get" id="$hBdee}oq43^UJcTz,;u"><field name="VAR" id="yv+gPAZd/8:NDOnmJ~uf">gridSearch</field></block></value><value name="ADD1"><block type="variables_get" id="+Q~gufbZ65hxF*8)q#$N"><field name="VAR" id="_z/H/tTnOYS1Lla2~;*0">X</field></block></value><value name="ADD2"><block type="varDoMethod" id="scikt5[Gq5cuXNTzF=]k"><field name="VAR" id="hxqH|@PJU~wrI]t*tHgd">np</field><field name="MEMBER">ravel</field><data>np:ravel</data><value name="INPUT"><block type="lists_create_with" id="=Mr6X5_/I`ansR9VIxbf"><mutation items="1"></mutation><value name="ADD0"><block type="variables_get" id="}VKVKM8x5o/F1Gu~)4Q$"><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field></block></value></block></value></block></value><value name="ADD3"><block type="dummyOutputCodeBlock" id="py$KuG=7@vxJRP2HGoIk"><field name="CODE">cv=10</field></block></value></block></value></block></value></block></next></block></xml><jupyter_output><empty_output><jupyter_text>Evaluate the model We can get the accuracy by comparing the predictions to *all* of `Y`:- `with metrics do accuracy_score using` a list containing    - `Y`    - `predictions`<jupyter_code>print(metrics.accuracy_score(Y, predictions))

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="jpz]0=,hLYv~rN^#0dFO">metrics</variable><variable id="(`;mrW|63Vww]$wlV9+1">predictions</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="text_print" id="u8M]eo-vRBhX!L/a@4+J" x="-102" y="208"><value name="TEXT"><shadow type="text" id=")gVsrXo+bq|?9m0cIU/Y"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="0B;?~qgT4?Xe8I|O=P)%"><field name="VAR" id="jpz]0=,hLYv~rN^#0dFO">metrics</field><field name="MEMBER">accuracy_score</field><data>metrics:</data><value name="INPUT"><block type="lists_create_with" id="qe7YbU#WU/F|Iuks0C:c"><mutation items="2"></mutation><value name="ADD0"><block type="variables_get" id="m^Z6y-8@L:s!oS)`%iJW"><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field></block></value><value name="ADD1"><block type="variables_get" id="|3h,TPit9wg+a;bR[r$P"><field name="VAR" id="(`;mrW|63Vww]$wlV9+1">predictions</field></block></value></block></value></block></value></block></xml><jupyter_output>0.9533333333333334<jupyter_text>And similarly we can get the recall and precision using all of `Y`:- `print with metrics do classification_report using` a list containing    - `Y`    - `predictions`<jupyter_code>print(metrics.classification_report(Y, predictions))

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="jpz]0=,hLYv~rN^#0dFO">metrics</variable><variable id="(`;mrW|63Vww]$wlV9+1">predictions</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="text_print" id="jFo8knaDn;JRZ=Fj7Tzo" x="57" y="316"><value name="TEXT"><shadow type="text" id="QWuxwm12CzEZ-r2sdwCR"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="(y`#h=AgNH6X+ahL$2L$"><field name="VAR" id="jpz]0=,hLYv~rN^#0dFO">metrics</field><field name="MEMBER">classification_report</field><data>metrics:classification_report</data><value name="INPUT"><block type="lists_create_with" id="A/#8KhfiGd[(@p,uaFFl"><mutation items="2"></mutation><value name="ADD0"><block type="variables_get" id="fQYj?Tz[y/t|%9~{2Pf["><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field></block></value><value name="ADD1"><block type="variables_get" id="mX(20KX!C)d_,/ddeysr"><field name="VAR" id="(`;mrW|63Vww]$wlV9+1">predictions</field></block></value></block></value></block></value></block></xml><jupyter_output>precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        50
  versicolor       0.94      0.92      0.93        50
   virginica       0.92      0.94      0.93        50

    accuracy                           0.95       150
   macro avg       0.95      0.95      0.95       150
weighted avg       0.95      0.95      0.95       150<jupyter_text>In this case, likely because the classification problem is so easy, there is no improvement between Gaussian naive Bayes and our hyperparameter-tuned SVM. HyperparameterInterestingly, the nested crossvalidation approach in `sklearn` does not return the best hyperparameter values found.The simplest option is to discover the hyperparameter values using all the data, but then only report the performance using nested crossvalidation above.Alternatively, one could write custom code to calculate the optimal hyperparameter values for each fold.To do the simple option:- `with gridSearch do fit` using a list containing    - `X`    - `with np do ravel using` a list containing `Y`<jupyter_code>gridSearch.fit(X, np.ravel(Y))

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="yv+gPAZd/8:NDOnmJ~uf">gridSearch</variable><variable id="_z/H/tTnOYS1Lla2~;*0">X</variable><variable id="hxqH|@PJU~wrI]t*tHgd">np</variable><variable id="o8#tp%!,_Okj2aVwX(DL">Y</variable></variables><block type="varDoMethod" id="[Jz*%r,%!;ZE1O65v:d`" x="-7" y="188"><field name="VAR" id="yv+gPAZd/8:NDOnmJ~uf">gridSearch</field><field name="MEMBER">fit</field><data>gridSearch:fit</data><value name="INPUT"><block type="lists_create_with" id="FEZ,H}LeT1x1Dt5L@)=?"><mutation items="2"></mutation><value name="ADD0"><block type="variables_get" id="Kj)ny|}Jqd@|%4@l[`RZ"><field name="VAR" id="_z/H/tTnOYS1Lla2~;*0">X</field></block></value><value name="ADD1"><block type="varDoMethod" id="C^H}tyXYLZmDoM2mZXRx"><field name="VAR" id="hxqH|@PJU~wrI]t*tHgd">np</field><field name="MEMBER">ravel</field><data>np:ravel</data><value name="INPUT"><block type="lists_create_with" id="!Kva0Uuu6Ux2y3,h-PG1"><mutation items="1"></mutation><value name="ADD0"><block type="variables_get" id="wc7#(kzv$@k2^J@]QoTC"><field name="VAR" id="o8#tp%!,_Okj2aVwX(DL">Y</field></block></value></block></value></block></value></block></value></block></xml><jupyter_output><empty_output><jupyter_text>Now the best hyperparameter values can be displayed:    - freestyle `gridSearch.best_params_`<jupyter_code>gridSearch.best_params_

#<xml xmlns="https://developers.google.com/blockly/xml"><block type="dummyOutputCodeBlock" id="-l+|(Er0I~hx@x,VxdlQ" x="-7" y="10"><field name="CODE">gridSearch.best_params_</field></block></xml><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16823020025257082, 'fraction_numerical': 0.02369354741716279, 'mean_word_length': 6.249128160418483, 'pattern_counts': {'":': 1, '<': 780, '<?xml version=': 0, '>': 780, 'https://': 17, 'lorem ipsum': 0, 'www.': 1, 'xml': 60}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '223236', 'license': 'Apache-2.0', 'n_tokens_mistral': 11425, 'n_tokens_neox': 10926, 'n_words': 2517, 'path': 'Crossvalidation.ipynb', 'repo_name': 'memphis-iis/datawhys-content-notebooks', 'size': '33267'}