<jupyter_start><jupyter_text>Create entry points to spark<jupyter_code>from pyspark import SparkContext
sc = SparkContext(master = 'local')

from pyspark.sql import SparkSession
spark = SparkSession.builder \
          .appName("Python Spark SQL basic example") \
          .config("spark.some.config.option", "some-value") \
          .getOrCreate()<jupyter_output><empty_output><jupyter_text>Linear regression without cross-valiation  Import data<jupyter_code>ad = spark.read.csv('data/Advertising.csv', header=True, inferSchema=True)
ad.show(5)<jupyter_output>+-----+-----+---------+-----+
|   TV|Radio|Newspaper|Sales|
+-----+-----+---------+-----+
|230.1| 37.8|     69.2| 22.1|
| 44.5| 39.3|     45.1| 10.4|
| 17.2| 45.9|     69.3|  9.3|
|151.5| 41.3|     58.5| 18.5|
|180.8| 10.8|     58.4| 12.9|
+-----+-----+---------+-----+
only showing top 5 rows<jupyter_text>Transform data structure<jupyter_code>from pyspark.ml.linalg import Vectors
ad_df = ad.rdd.map(lambda x: [Vectors.dense(x[0:3]), x[-1]]).toDF(['features', 'label'])
ad_df.show(5)<jupyter_output>+-----------------+-----+
|         features|label|
+-----------------+-----+
|[230.1,37.8,69.2]| 22.1|
| [44.5,39.3,45.1]| 10.4|
| [17.2,45.9,69.3]|  9.3|
|[151.5,41.3,58.5]| 18.5|
|[180.8,10.8,58.4]| 12.9|
+-----------------+-----+
only showing top 5 rows<jupyter_text>Build linear regression model<jupyter_code>from pyspark.ml.regression import LinearRegression
lr = LinearRegression(featuresCol = 'features', labelCol = 'label')<jupyter_output><empty_output><jupyter_text>Fit the model<jupyter_code>lr_model = lr.fit(ad_df)<jupyter_output><empty_output><jupyter_text>Prediction<jupyter_code>pred = lr_model.transform(ad_df)
pred.show(5)<jupyter_output>+-----------------+-----+------------------+
|         features|label|        prediction|
+-----------------+-----+------------------+
|[230.1,37.8,69.2]| 22.1| 20.52397440971517|
| [44.5,39.3,45.1]| 10.4|12.337854820894362|
| [17.2,45.9,69.3]|  9.3|12.307670779994238|
|[151.5,41.3,58.5]| 18.5| 17.59782951168913|
|[180.8,10.8,58.4]| 12.9|13.188671856831299|
+-----------------+-----+------------------+
only showing top 5 rows<jupyter_text>Module evaluation<jupyter_code>from pyspark.ml.evaluation import RegressionEvaluator 
evaluator = RegressionEvaluator(predictionCol='prediction', labelCol='label')
evaluator.setMetricName('r2').evaluate(pred)<jupyter_output><empty_output><jupyter_text>Compare results with RThe comparison below shows that the linear regression analyses from pyspark and R obtained very close results. ```{r} intercept and coefficients from Radvertise = read.csv('data/Advertising.csv', header = TRUE)lr_ad = lm(Sales~., data = advertise)lr_ad$coefficients (Intercept)           TV        Radio    Newspaper  2.938889369  0.045764645  0.188530017 -0.001037493  intercept and coefficents from pysparklr_model.intercept2.9388893694594134lr_model.coefficientsDenseVector([0.0458, 0.1885, -0.001]) R squared from Rsummary(lr_ad)$r.squared0.8972106 R squared from pysparkevaluator.evaluate(ad_pred, {evaluator.metricName: "r2"})0.897210638178952```  Linear regression with cross-validation  Training and test datasets<jupyter_code>training, test = ad_df.randomSplit([0.8, 0.2], seed=123)<jupyter_output><empty_output><jupyter_text>Build cross-validation model<jupyter_code>##=====build cross valiation model======

# estimator
lr = LinearRegression(featuresCol = 'features', labelCol = 'label')

# parameter grid
from pyspark.ml.tuning import ParamGridBuilder
param_grid = ParamGridBuilder().\
    addGrid(lr.regParam, [0, 0.5, 1]).\
    addGrid(lr.elasticNetParam, [0, 0.5, 1]).\
    build()
    
# evaluator
evaluator = RegressionEvaluator(predictionCol='prediction', labelCol='label', metricName='r2')

# cross-validation model
from pyspark.ml.tuning import CrossValidator
cv = CrossValidator(estimator=lr, estimatorParamMaps=param_grid, evaluator=evaluator, numFolds=4)<jupyter_output><empty_output><jupyter_text>Fit cross-validation model<jupyter_code>cv_model = cv.fit(training)<jupyter_output><empty_output><jupyter_text>Prediction<jupyter_code>pred_training_cv = cv_model.transform(training)
pred_test_cv = cv_model.transform(test)<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code># performance on training data
evaluator.setMetricName('r2').evaluate(pred_training_cv)
# performance on test data
evaluator.setMetricName('r2').evaluate(pred_test_cv)<jupyter_output><empty_output><jupyter_text>Intercept and coefficients<jupyter_code>print('Intercept: ', cv_model.bestModel.intercept, "\n",
     'coefficients: ', cv_model.bestModel.coefficients)<jupyter_output>Intercept:  3.075068686285647 
 coefficients:  [0.0465074974309,0.180854522465,-0.00107520549074]<jupyter_text>Get parameter values from the best model Parameters can be extracted by calling the java property.<jupyter_code>print('best regParam: ' + str(cv_model.bestModel._java_obj.getRegParam()) + "\n" +
     'best ElasticNetParam:' + str(cv_model.bestModel._java_obj.getElasticNetParam()))<jupyter_output>best regParam: 0.0
best ElasticNetParam:0.0
{'fraction_non_alphanumeric': 0.18849714398266693, 'fraction_numerical': 0.08981682095725822, 'mean_word_length': 4.53762268266085, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '345831', 'license': 'MIT', 'n_tokens_mistral': 2132, 'n_tokens_neox': 1858, 'n_words': 293, 'path': 'notebooks/06-machine-learning/regression/linear-regression.ipynb', 'repo_name': 'peymanyousefianasml/learning-apache-spark', 'size': '5077'}