<jupyter_start><jupyter_text>Diabetes - Linear RegressionThis notebook models the relation between the Body mass index and a quantitative measure of disease progression one year after baseline as a Linear Regression.Instructions about Spark installation on [this medium post](https://medium.com/@roshinijohri/spark-with-jupyter-notebook-on-macos-2-0-0-and-higher-c61b971b5007)<jupyter_code>import os
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
%matplotlib inline

exec(open(os.path.join(os.environ["SPARK_HOME"], 'python/pyspark/shell.py')).read())<jupyter_output>Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.3
      /_/

Using Python version 3.7.1 (default, Nov 28 2018 11:51:54)
SparkSession available as 'spark'.<jupyter_text>Load the datasetDownload the data file from [NSCU](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.tab.txt) and plot some aspects of it.__Attribute Information:__First 10 columns are numeric predictive values:- Age- Sex- Body mass index- Average blood pressure- S1- S2- S3- S4- S5- S6(six blood serum measurements)__Target:__Column 11 is a quantitative measure of disease progression one year after baseline<jupyter_code>df = pd.read_csv('https://www4.stat.ncsu.edu/~boos/var.select/diabetes.tab.txt', delimiter='\t')
df.info()
X = df.BMI # Body mass index
y = df.Y
plt.scatter(X, y,  color='black')<jupyter_output><empty_output><jupyter_text>Create a Spark DataframeLoad the pandas data frame as a SparkDataframe<jupyter_code>diabetes = spark.createDataFrame(df)
diabetes.columns
diabetes.show(3)<jupyter_output>+---+---+----+-----+---+-----+----+---+------+---+---+
|AGE|SEX| BMI|   BP| S1|   S2|  S3| S4|    S5| S6|  Y|
+---+---+----+-----+---+-----+----+---+------+---+---+
| 59|  2|32.1|101.0|157| 93.2|38.0|4.0|4.8598| 87|151|
| 48|  1|21.6| 87.0|183|103.2|70.0|3.0|3.8918| 69| 75|
| 72|  2|30.5| 93.0|156| 93.6|41.0|4.0|4.6728| 85|141|
+---+---+----+-----+---+-----+----+---+------+---+---+
only showing top 3 rows<jupyter_text>Transform Spark Dataframe on VectorAssembler<jupyter_code>from pyspark.ml.feature import VectorAssembler

vectorAssembler = VectorAssembler(inputCols = ['AGE', 'SEX', 'BMI', 'BP', 'S1', 'S2', 'S3', 'S4', 'S5', 'S6'], outputCol = 'features')
vDiabetes = vectorAssembler.transform(diabetes)
vDiabetes = vDiabetes.select(['features', 'Y'])
vDiabetes.show(3)<jupyter_output>+--------------------+---+
|            features|  Y|
+--------------------+---+
|[59.0,2.0,32.1,10...|151|
|[48.0,1.0,21.6,87...| 75|
|[72.0,2.0,30.5,93...|141|
+--------------------+---+
only showing top 3 rows<jupyter_text>Split data on train and test sets<jupyter_code>splits = vDiabetes.randomSplit([0.7, 0.3])
train = splits[0]
test = splits[1]
train.show(3)
from pyspark.sql.functions import udf
from pyspark.sql.types import FloatType

# train.select('features').collect()
firstelement=udf(lambda v:float(v[2]),FloatType())
X = train.select(firstelement('features'))
X = [float(row[0]) for row in X.collect()]
y = [float(row['Y']) for row in train.collect()]

xTest = test.select(firstelement('features'))
xTest = [float(row[0]) for row in xTest.collect()]
yTest = [float(row['Y']) for row in test.collect()]

plt.scatter(X, y,  color='black')
plt.scatter(xTest, yTest,  color='blue')<jupyter_output><empty_output><jupyter_text>Create the Linear Regression model and fit it<jupyter_code>from pyspark.ml.regression import LinearRegression
lr = LinearRegression(featuresCol = 'features', labelCol='Y', maxIter=30, regParam=0.3, elasticNetParam=0.8)

# Fit the model
lrModel = lr.fit(train)
print("Coefficients: " + str(lrModel.coefficients))
print("Intercept: " + str(lrModel.intercept))
trainingSummary = lrModel.summary
print("RMSE: %f" % trainingSummary.rootMeanSquaredError)
print("r2: %f" % trainingSummary.r2)
train.describe().show()
test.show(5)
from pyspark.ml.evaluation import RegressionEvaluator

predictions = lrModel.transform(test)
predictions.select("prediction","Y","features").show(5)


evaluator = RegressionEvaluator(predictionCol="prediction", \
                 labelCol="Y",metricName="r2")
print("R Squared (R2) on test data = %g" % evaluator.evaluate(predictions))
testResult = lrModel.evaluate(test)
print("Root Mean Squared Error (RMSE) on test data = %g" % testResult.rootMeanSquaredError)
print("numIterations: %d" % trainingSummary.totalIterations)
print("objectiveHistory: %s" % str(trainingSummary.objectiveHistory))
trainingSummary.residuals.show()<jupyter_output>numIterations: 31
objectiveHistory: [0.5000000000000002, 0.4441897409829618, 0.27703015222532285, 0.2699412925712932, 0.2539047955403203, 0.250753362377854, 0.25035686153417186, 0.2502373420672937, 0.25012550662010646, 0.2499749229631131, 0.24983175646265973, 0.24975550053664866, 0.24953094143878754, 0.24946033599615822, 0.24945155346519418, 0.24939447740404214, 0.2493748177303759, 0.24936248811162115, 0.24935152021664814, 0.24934229178837894, 0.24933117217972212, 0.2493247731134857, 0.2493040505717619, 0.24928276388626333, 0.24927456281624596, 0.24925970220961036, 0.24925614117807965, 0.24924697456007064, 0.24924477527685432, 0.24923852518395523, 0.24923206427225456]
+-------------------+
|          residuals|
+-------------------+
|  37.86274834799025|
| 17.208388218065068|
|-23.104822433034713|
|   57.5790923542437|
| 118.43980740917993|
| -5.091818595569123|
|  157.2217650821094|
|  39.72266937219575|
| -45.19389693008253|
|-39.434108202210155|
|  33.31209552215745|
|-13.8347809688[...]<jupyter_text>Constructs Simple Linear RegressionConsiders just the BMI feature to predict Y.<jupyter_code>from pyspark.ml.feature import VectorAssembler

vectorAssembler = VectorAssembler(inputCols = ['BMI'], outputCol = 'features')
vDiabetes = vectorAssembler.transform(diabetes)
vDiabetes = vDiabetes.select(['features', 'Y'])
vDiabetes.show(3)
sSplits = vDiabetes.randomSplit([0.7, 0.3])
sTrain = sSplits[0]
sTest = sSplits[1]
sTest.show(5)
from pyspark.ml.regression import LinearRegression
lrSimple = LinearRegression(featuresCol='features', labelCol='Y', maxIter=10, regParam=0.3, elasticNetParam=0.8)

# Fit the model
lrSimpleModel = lrSimple.fit(sTrain)
print("Coefficients: " + str(lrSimpleModel.coefficients))
print("Intercept: " + str(lrSimpleModel.intercept))
summary = lrSimpleModel.summary
print("RMSE: %f" % summary.rootMeanSquaredError)
print("r2: %f" % summary.r2)
from pyspark.ml.evaluation import RegressionEvaluator
print(sTest.count)
sPredictions = lrSimpleModel.transform(sTest)
print(sPredictions.count)
sPredictions.select("prediction","Y","features").show(5)

sEvaluator = RegressionEvaluator(predictionCol="prediction", \
                 labelCol="Y",metricName="r2")
print("R Squared (R2) on test data = %g" % sEvaluator.evaluate(sPredictions))
from pyspark.sql.functions import udf
from pyspark.sql.types import FloatType

firstelement=udf(lambda v:float(v[0]),FloatType())
xsTest = sTest.select(firstelement('features'))
xsTest = [float(row[0]) for row in xsTest.collect()]
print(len(xsTest))

ysTest = [float(row['Y']) for row in sTest.collect()]
print(len(ysTest))
yPred = [float(row["prediction"]) for row in sPredictions.collect()]
print(len(yPred))
plt.scatter(xsTest, ysTest,  color='blue')
plt.plot(xsTest, yPred, color='black', linewidth=3)
sTest = lrSimpleModel.evaluate(sTest)
print("Root Mean Squared Error (RMSE) on test data = %g" % sTest.rootMeanSquaredError)<jupyter_output>Root Mean Squared Error (RMSE) on test data = 64.8333
{'fraction_non_alphanumeric': 0.17495689083432817, 'fraction_numerical': 0.12959278418888448, 'mean_word_length': 4.890625, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '597566', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3447, 'n_tokens_neox': 2816, 'n_words': 525, 'path': 'notebooks/Linear Regression/Diabetes-Linear_Reg-Spark.ipynb', 'repo_name': 'JPMagalhaesCESAR/MLIntro', 'size': '7539'}