<jupyter_start><jupyter_text>Spark ML - Linear Regression (Adv Example)Now that you've completed the documentation example, it's good to go through something a little more complex. This tutorial's materials have been adapted from <NAME>. Find more information here: - https://towardsdatascience.com/building-a-linear-regression-with-pyspark-and-mllib-d065c3ba246a- https://github.com/susanli2016/PySpark-and-MLlib.Objective: We're using a dataset from Kaggle to predict housing prices in Boston. We'll explore the data, visualise it, create a vector, build a regression model and evaluate it.<jupyter_code># Section must be included at the beginning of each new notebook. Remember to change the app name.
# If you're using VirtualBox, change the below to '/home/user/spark-2.1.1-bin-hadoop2.7'
import findspark
findspark.init('/home/ubuntu/spark-2.1.1-bin-hadoop2.7')
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('linear_regression_adv').getOrCreate()

# If you're getting an error with numpy, please type 'sudo pip3 install numpy --user' into the console.
# If you're getting an error with another package, type 'sudo pip3 install PACKAGENAME --user'. 
# Replace PACKAGENAME with the relevant package (such as pandas, etc).
from pyspark.ml.regression import LinearRegression

# First, let's import the data. Note that we can infer the schema as it's a CSV file.
df = spark.read.csv("Datasets/boston_housing_data.csv",inferSchema=True,header=True)
# Let's explore. Here's the first row of the data.
print(df.head())

# And the entire data structure. 
df.printSchema()<jupyter_output>Row(crim=0.00632, zn=18.0, indus=2.31, chas=0, nox=0.538, rm=6.575, age=65.2, dis=4.09, rad=1, tax=296, ptratio=15.3, b=396.9, lstat=4.98, medv=24.0)
root
 |-- crim: double (nullable = true)
 |-- zn: double (nullable = true)
 |-- indus: double (nullable = true)
 |-- chas: integer (nullable = true)
 |-- nox: double (nullable = true)
 |-- rm: double (nullable = true)
 |-- age: double (nullable = true)
 |-- dis: double (nullable = true)
 |-- rad: integer (nullable = true)
 |-- tax: integer (nullable = true)
 |-- ptratio: double (nullable = true)
 |-- b: double (nullable = true)
 |-- lstat: double (nullable = true)
 |-- medv: double (nullable = true)<jupyter_text>Feature Information- CRIM — per capita crime rate by town.- ZN — proportion of residential land zoned for lots over 25,000 sq.ft.- INDUS — proportion of non-retail business acres per town.- CHAS — Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).- NOX — nitrogen oxides concentration (parts per 10 million).- RM — average number of rooms per dwelling.- AGE — proportion of owner-occupied units built prior to 1940.- DIS — weighted mean of distances to five Boston employment centres.- RAD — index of accessibility to radial highways.- TAX — full-value property-tax rate per \$10,000.- PTRATIO — pupil-teacher ratio by town.- BLACK — 1000(Bk — 0.63)² where Bk is the proportion of blacks by town.- LSTAT — lower status of the population (percent).- MEDV — median value of owner-occupied homes in $1000s. This is the target variable.<jupyter_code># Now that we understand the data's features, let's use a Python package to neatly describe the data.
import pandas as pd
df.describe().toPandas().transpose()<jupyter_output><empty_output><jupyter_text>Setting Up a DataFrame for Machine Learning (MLlib)We need to do a few things before Spark can accept this data for machine learning. First of all, it needs to be in the form of two columns: label and features. Unlike the documentation example, this data is messy. We'll need to combine all of the features into a single vector. VectorAssembler simplifies the process.<jupyter_code># Import VectorAssembler and Vectors
from pyspark.ml.feature import VectorAssembler

# The input columns are the feature column names, and the output column is what you'd like the new column to be named. 
vector_assembler = VectorAssembler(inputCols = ['crim', 'zn', 'indus', 'chas', 'nox', 'rm', 'age', 'dis', 'rad', 'tax', 'ptratio', 'b', 'lstat'], outputCol = 'features')

# Now that we've created the assembler variable, let's actually transform the data.
vector_output = vector_assembler.transform(df)

# Using print schema, you see that the features output column has been added. 
vector_output.printSchema()

# You can see that the features column is a DenseVector that combines the various features as expected.
vector_output.head(1)
# Because the features have been combined into one vector, we no longer need them. Below we select the features and label.
vector_output = vector_output.select(['features', 'medv'])

# You can see that the dataframe now only contains two columns. 
print(vector_output.head(1))
vector_output.show(3)
# Let's do a randomised 70/30 split. Remember, you should explain why you chose a particular split. 
train_data,test_data = vector_output.randomSplit([0.7,0.3])

# Let's see our training data.
train_data.describe().show()

# And our testing data.
test_data.describe().show()<jupyter_output>+-------+------------------+
|summary|              medv|
+-------+------------------+
|  count|               347|
|   mean|22.520461095100888|
| stddev| 9.055746543025418|
|    min|               5.0|
|    max|              50.0|
+-------+------------------+

+-------+------------------+
|summary|              medv|
+-------+------------------+
|  count|               159|
|   mean|22.559748427672957|
| stddev| 9.527396336301386|
|    min|               5.0|
|    max|              50.0|
+-------+------------------+<jupyter_text>Linear RegressionNow we can create a Linear Regression Model instance. Because the feature column is named 'features', we don't have to worry about it. However, as the labelCol isn't the default name, we have to specify it's name (medv).<jupyter_code># Importing the LR package.
from pyspark.ml.regression import LinearRegression

# Instantiate the instance.
lr = LinearRegression(featuresCol='features', labelCol='medv')

# Fit the training data.
lr_model = lr.fit(train_data)

# Print the coefficients.
print("Coefficients: " + str(lr_model.coefficients))

# Print the intercept.
print("Intercept: " + str(lr_model.intercept) + "\n")

# Summarise the model and print out some evaluation metrics.
training_summary = lr_model.summary

# Print RMSE. 
print("RMSE: " + str(training_summary.rootMeanSquaredError))

# Print R2.
print("R2: " + str(training_summary.r2))
train_data.describe().show()<jupyter_output>+-------+------------------+
|summary|              medv|
+-------+------------------+
|  count|               347|
|   mean|22.520461095100888|
| stddev| 9.055746543025418|
|    min|               5.0|
|    max|              50.0|
+-------+------------------+<jupyter_text>RMSE measures the differences between predicted values and actual values. However, RMSE alone is meaningless until we compare with the actual "medv" value, such as mean, min and max. After such comparison, our RMSE looks pretty good.R2 indicates that our model can explain approximately 70% of the variability in median house value (medv).  Evaluating the Model using the Testing Set<jupyter_code># Let's evaluate the model against the test data.
test_results = lr_model.evaluate(test_data)

# And print the RMSE/R2. As expected, our RMSE and R2 are slightly worse when applying the testing set.
print("RMSE on test data: " + str(test_results.rootMeanSquaredError))
print("R2 on test data: " + str(test_results.r2))<jupyter_output>RMSE on test data: 4.626537055363597
R2 on test data: 0.7626967626458543
{'fraction_non_alphanumeric': 0.1313503553566728, 'fraction_numerical': 0.03606212161095025, 'mean_word_length': 3.6082474226804124, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '258947', 'license': 'MIT', 'n_tokens_mistral': 2523, 'n_tokens_neox': 2292, 'n_words': 862, 'path': 'Spark ML - Linear Regression (Adv Example).ipynb', 'repo_name': 'vwen01/PySpark_IncomeDataset', 'size': '7600'}