<jupyter_start><jupyter_text>Decision Trees with PySpark This notebook creates and measures a [Decision Tree](https://spark.apache.org/docs/2.2.0/mllib-decision-tree.html) model with PySpark.* Method: Decision Tree* Dataset: MLlib Sample Data  Imports<jupyter_code>from os import environ
# Set SPARK_HOME
environ["SPARK_HOME"] = "/home/students/spark-2.2.0"

import findspark
findspark.init()

import numpy as np

from pyspark import SparkContext
from pyspark.sql import SQLContext

from pyspark.mllib.tree import DecisionTree
from pyspark.mllib.util import MLUtils

import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Get Some Context<jupyter_code># Create a SparkContext and a SQLContext context to use
sc = SparkContext(appName="Decision Tree Classification with Spark")
sqlContext = SQLContext(sc)<jupyter_output><empty_output><jupyter_text>Load and Prepare the Data<jupyter_code>DATA_FILE = "/home/students/data/mllib/sample_libsvm_data.txt"
# Load the training data
data = MLUtils.loadLibSVMFile(sc, path=DATA_FILE)
# Show one of the records
data.take(1)
# Create train and test datasets
train, test = data.randomSplit([0.8, 0.2], 42)
print(train.count(), test.count())<jupyter_output><empty_output><jupyter_text>Fit a Decision Tree ModelArguments* numClasses: number of classes* categoricalFeaturesInfo: specifies which features are categorical and how many categorical values each of those features can take.* impurity: a measure of the homogeneity of the labels at the node; options: gini and entropy* maxDepth: maximum depth of the tree* maxBins: number of bins used when discretizing continuous features.  * More = allows the algo to consider more split candidtates and make fine-grained split decisions; increases computation<jupyter_code>model = DecisionTree.trainClassifier(train,
                                     numClasses=2,
                                     categoricalFeaturesInfo={},
                                     impurity='gini',
                                     maxDepth=5,
                                     maxBins=32)<jupyter_output><empty_output><jupyter_text>Create Predictions<jupyter_code>predictions = model.predict(test.map(lambda x: x.features))
labels_and_predictions = test.map(lambda lp: lp.label).zip(predictions)
# Convert labels_and_predictions and the test RDD to dataframes
lp_df = sqlContext.createDataFrame(labels_and_predictions, ["label", "predicted"])
test_df = sqlContext.createDataFrame(test, ["features", "label"])

# Make sure they have the same number of records
print(lp_df.count(), test_df.count())
# Check the dataframes
print(lp_df.show(5))
print(test_df.show(5))
# Create a plot to compare the actuals (labels) and predictions
actuals = lp_df.rdd.map(lambda r: r.label).collect()
predictions = lp_df.rdd.map(lambda r: float(r.predicted)).collect()


fig = plt.figure(figsize=(10,5))
plt.scatter(actuals, predictions)
plt.xlabel("Actuals")
plt.ylabel("Predictions")
plt.title("Actuals vs. Predictions")
plt.show()<jupyter_output><empty_output><jupyter_text>Model Evaluation  Training ErrorCalculate the training error<jupyter_code>training_error = labels_and_predictions.filter(lambda lp: lp[0] != lp[1]).count() / float(test.count())
print("Training Error = %.2f" % training_error)<jupyter_output><empty_output><jupyter_text>View the Tree<jupyter_code>print(model.toDebugString())<jupyter_output><empty_output><jupyter_text>Shut it Down<jupyter_code>sc.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10125499144324016, 'fraction_numerical': 0.00713063320022818, 'mean_word_length': 5.099130434782609, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '184103', 'license': 'Apache-2.0', 'n_tokens_mistral': 1121, 'n_tokens_neox': 1080, 'n_words': 273, 'path': 'code/day_6/5 - Decision Trees with PySpark.ipynb', 'repo_name': 'anjijava16/data-analytics-machine-learning-big-data', 'size': '3506'}