<jupyter_start><jupyter_text>" 3.4.1 Decision TreesAfter completing this lesson you should be able to:* Understand the pipelines API for decision trees* Describe Pipeline's input and output columns * Perform classification and regression with decision trees * Understand and use decision trees parameters  Decision Trees A decision tree is an effective non-parametric modeling method for different types of data. It can handle both continuous and categorical data. It can also help us identify markers that split our categories. * Popular method for classification and regression* Easy to interoret* Handle categorical features* Extend to multiclass classification* Do NOT require feature scaling* They capture non-linearities and feature interactions MLlib's implementation MLlib's implementation of Decision Trees supports both binary and multi-class classification as well as regression. It also supports both continuous and categorical features. In MLlib the data is partitioned by rows allowing distributed training of Decision Trees. * Supports binary and multiclass classification* Supports regression* Supports continuous and categorical features * Partitions data by rows, allowing distributed training* Used by the Pipelines API for Decision TreesThis implementation is then used by the Pipelines API for Decision Trees which I'll introduce in this lesson.  Spark.ML API for Decision Trees* More functionalities in the original MLlib API * Separation of Decision Trees for classification and regression* Use of DF metadata to distinguish between continuous and categorical features* For classification trees	* class conditional probabilities, that is, predicted probabilities of each class, made available	* estimates of feature importanceThe Spark.ML API for decision trees contains more functionalities in the original MLlib API.There is a separation of decision trees the classification and regression.We use DataFrames meta data to distinguish between continuous and categorical features  Inputs and Outputs Now let's take a look at the inputs taken and the output produced by decision trees in the Pipelines API. As inputs, there are the label and feature columns. If its corresponding parameter names are omitted the API will look for its defaults, label and features in the import DataFrame. As outputs there are the prediction, rawPrediction and probability columns, where the last two only apply for classification trees. If its corresponding parameter names are omitted the API will append these columns with its default names to the resulting DataFrame.Notice that if I don't want to give a column to be appended I just have to set its name as an empty string using the appropriate parameter name.  Loading Data<jupyter_code>import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().getOrCreate()
import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.ml.classification.DecisionTreeClassifier
import org.apache.spark.ml.classification.DecisionTreeClassificationModel
import sys.process._
"wget https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/SC0105EN/data/sample_libsvm_data.txt  -P /resources/data/"!
val dtC = new DecisionTreeClassifier().setLabelCol("indexedLabel").setFeaturesCol("indexedFeatures")

import org.apache.spark.ml.Pipeline
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.ml.feature.{StringIndexer, IndexToString, VectorIndexer}
import org.apache.spark.mllib.util.MLUtils.{
  convertVectorColumnsFromML => fromML,
  convertVectorColumnsToML => toML
}

val data = toML(MLUtils.loadLibSVMFile(sc, "/resources/data/sample_libsvm_data.txt").toDF)
data.show()<jupyter_output>+-----+--------------------+
|label|            features|
+-----+--------------------+
|  0.0|(692,[127,128,129...|
|  1.0|(692,[158,159,160...|
|  1.0|(692,[124,125,126...|
|  1.0|(692,[152,153,154...|
|  1.0|(692,[151,152,153...|
|  0.0|(692,[129,130,131...|
|  1.0|(692,[158,159,160...|
|  1.0|(692,[99,100,101,...|
|  0.0|(692,[154,155,156...|
|  0.0|(692,[127,128,129...|
|  1.0|(692,[154,155,156...|
|  0.0|(692,[153,154,155...|
|  0.0|(692,[151,152,153...|
|  1.0|(692,[129,130,131...|
|  0.0|(692,[154,155,156...|
|  1.0|(692,[150,151,152...|
|  0.0|(692,[124,125,126...|
|  0.0|(692,[152,153,154...|
|  1.0|(692,[97,98,99,12...|
|  1.0|(692,[124,125,126...|
+-----+--------------------+
only showing top 20 rows<jupyter_text>This is a sample data set that comes with Spark. If you didn't get Spark from sources, then you probably don't have it. From a Spark notebook you can get it from the shell command `wget`; remember this will not work on Microsoft Windows boxes. Then we can load the data into a `DataFrame` using the `loadLibSVMFile()` function and then calling `toDF` on it.  Creating the Tree Model<jupyter_code>val labelIndexer = new StringIndexer().setInputCol("label").setOutputCol("indexedLabel").fit(data)
val labelConverter = new IndexToString().setInputCol("prediction").setOutputCol("predictedLabel").setLabels(labelIndexer.labels)
val featureIndexer = new VectorIndexer().setInputCol("features").setOutputCol("indexedFeatures").setMaxCategories(4).fit(data)
val pipelineClass = new Pipeline().setStages(Array(labelIndexer, featureIndexer, dtC, labelConverter))
val Array(trainingData, testData) = data.randomSplit(Array(0.7, 0.3))<jupyter_output><empty_output><jupyter_text>Next we set up our input and output columns and then fit the data for the `labelIndexer`. From that, we get the `labelConverter` from it. Then we set up a `featureIndexer` from `VectorIndexer`, which will perform a fit call on with our input data. Next we set up the pipeline of the data with the stages are derived from those values we just created and perform a random split to get our `trainingData` and `testData` values.  DecisionTreeClassifier I<jupyter_code>val modelClassifier = pipelineClass.fit(trainingData)
val treeModel = modelClassifier.stages(2).asInstanceOf[DecisionTreeClassificationModel]
println("Learned classification tree model:\n" + treeModel.toDebugString)<jupyter_output>Learned classification tree model:
DecisionTreeClassificationModel (uid=dtc_19955a85afcb) of depth 2 with 5 nodes
  If (feature 406 <= 22.0)
   If (feature 99 in {2.0})
    Predict: 0.0
   Else (feature 99 not in {2.0})
    Predict: 1.0
  Else (feature 406 > 22.0)
   Predict: 0.0<jupyter_text>To get our `DecisionTreeClassifier` we create a `ModelClassifier` by fitting our pipeline from the `trainingData` values and then we create the `treeModels` by casting the data to an instance of a Decision Tree Classification model and then print out the data. DecisionTreeClassifier II<jupyter_code>// Getting results from the model
val predictionsClass = modelClassifier.transform(testData)
predictionsClass.show()<jupyter_output>+-----+--------------------+------------+--------------------+-------------+-----------+----------+--------------+
|label|            features|indexedLabel|     indexedFeatures|rawPrediction|probability|prediction|predictedLabel|
+-----+--------------------+------------+--------------------+-------------+-----------+----------+--------------+
|  0.0|(692,[95,96,97,12...|         1.0|(692,[95,96,97,12...|   [0.0,28.0]|  [0.0,1.0]|       1.0|           0.0|
|  0.0|(692,[100,101,102...|         1.0|(692,[100,101,102...|   [0.0,28.0]|  [0.0,1.0]|       1.0|           0.0|
|  0.0|(692,[123,124,125...|         1.0|(692,[123,124,125...|   [0.0,28.0]|  [0.0,1.0]|       1.0|           0.0|
|  0.0|(692,[123,124,125...|         1.0|(692,[123,124,125...|   [0.0,28.0]|  [0.0,1.0]|       1.0|           0.0|
|  0.0|(692,[124,125,126...|         1.0|(692,[124,125,126...|   [0.0,28.0]|  [0.0,1.0]|       1.0|           0.0|
|  0.0|(692,[124,125,126...|         1.0|(692,[124,125,126...|   [0.0,28.0]|  [0[...]<jupyter_text>You may notice that the `Indexedlabel` is presented as 1, while label suggests 0, and vice versa. Please note the by  passing the model by the String Indexer and label converter, Spark has signed its own alias to the labels provided, hence the prediction is also represented in Spark's own alias. The predicted label, however, returns a string value that represents the original names of the labels in the dataset. DecisionTreeRegressor I<jupyter_code>import org.apache.spark.ml.regression.DecisionTreeRegressor
import org.apache.spark.ml.regression.DecisionTreeRegressionModel

val dtR = new DecisionTreeRegressor().setLabelCol("label").setFeaturesCol("indexedFeatures")

val pipelineReg = new Pipeline().setStages(Array(featureIndexer, dtR))<jupyter_output><empty_output><jupyter_text>The same goes for the decision tree `regressor`. In the case of regression, remember that my Pipeline had only two stages: the features indexer and the decision tree `regressor`.  DecisionTreeRegressor II<jupyter_code>val modelRegressor = pipelineReg.fit(trainingData)

val treeModel = modelRegressor.stages(1).asInstanceOf[DecisionTreeRegressionModel]

println("Learned regression tree model:\n" + treeModel.toDebugString)<jupyter_output>Learned regression tree model:
DecisionTreeRegressionModel (uid=dtr_563ae6fc9358) of depth 2 with 5 nodes
  If (feature 406 <= 22.0)
   If (feature 99 in {0.0,3.0})
    Predict: 0.0
   Else (feature 99 not in {0.0,3.0})
    Predict: 1.0
  Else (feature 406 > 22.0)
   Predict: 1.0<jupyter_text>Similarly to a `DecisionTreeClassifier` we get a `DecisionTreeRegressor` by creating a `modelRegressor` value from fitting our pipeline from the `trainingData` values. And, like before, we create the `treeModels` by casting the data to an instance of a Decision Tree Regression Model, and print out the data.  DecisionTreeRegressor III<jupyter_code>// Getting results from the model
val predictionsReg = modelRegressor.transform(testData)
predictionsReg.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13454324242117002, 'fraction_numerical': 0.061745919091554295, 'mean_word_length': 4.142857142857143, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '241557', 'license': 'MIT', 'n_tokens_mistral': 3337, 'n_tokens_neox': 2889, 'n_words': 974, 'path': 'Scala/3.4.1.ipynb', 'repo_name': 'verneh/DataSci', 'size': '9863'}