<jupyter_start><jupyter_text>Iris - Logistic RegressionThis example applies Logistic Regression in order to predict the type of iris plant based on sepal and petal length and width. Import some useful libraries and get Spark Context<jupyter_code>from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

sc= SparkContext()
sqlContext = SQLContext(sc)<jupyter_output><empty_output><jupyter_text>Load the database, convert it to Spark and split data into train and test sets[Iris Data Set](https://archive.ics.uci.edu/ml/datasets/iris)Attribute Information: 	- sepal length in cm- sepal width in cm- petal length in cm- petal width in cm- class:    - Iris-Setosa    - Iris-Versicolour    - Iris-Virginica<jupyter_code>pdf = pd.read_csv('iris.data')
pdf.columns = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species']
data = sqlContext.createDataFrame(pdf)
pdf.head()<jupyter_output><empty_output><jupyter_text>Data exploration<jupyter_code>pdf.describe()
data.printSchema()
data.show(3)
from pyspark.ml.feature import StringIndexer

indexer = StringIndexer(inputCol="species", outputCol="label")
indexed = indexer.fit(data).transform(data)
indexed.select(['label']).distinct().show()
from pyspark.ml.feature import VectorAssembler

vectorAssembler = VectorAssembler(inputCols = ['sepal_length', "sepal_width", "petal_length", "petal_width"], outputCol = 'features')
vData = vectorAssembler.transform(indexed)
vData = vData.select(['features', 'label'])
vData.show(3)
train, test = vData.randomSplit([0.7, 0.3])

print("Training Dataset Count: " + str(train.count()))
print("Test Dataset Count: " + str(test.count()))<jupyter_output>Training Dataset Count: 97
Test Dataset Count: 52<jupyter_text>Logistic Regression<jupyter_code>from pyspark.ml.classification import LogisticRegression

lr = LogisticRegression(featuresCol = 'features', labelCol = 'label', maxIter=10)
lrModel = lr.fit(train)
print(lrModel.coefficientMatrix)
predictions = lrModel.transform(test)
predictions.select('prediction', 'label', 'rawPrediction', 'probability').show(10)
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

evaluator = MulticlassClassificationEvaluator()
print('Test Area Under ROC', evaluator.evaluate(predictions))
print('Higher the AUC, better the model is at predicting')<jupyter_output>Test Area Under ROC 0.9616575375089308
Higher the AUC, better the model is at predicting
{'fraction_non_alphanumeric': 0.10988578180385979, 'fraction_numerical': 0.012209531311539977, 'mean_word_length': 7.115015974440895, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '501825', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 851, 'n_tokens_neox': 811, 'n_words': 211, 'path': 'notebooks/Logistic Regression/Iris-Logistic_Regression-Spark.ipynb', 'repo_name': 'JPMagalhaesCESAR/MLIntro', 'size': '2539'}