<jupyter_start><jupyter_text>Train / Test Split<jupyter_code>all_data = spark.read.format('libsvm').load('../data/sample_linear_regression_data.txt')
train, test = all_data.randomSplit([0.7, 0.3])
train.describe().show()
test.describe().show()<jupyter_output>+-------+-------------------+
|summary|              label|
+-------+-------------------+
|  count|                148|
|   mean| 0.3882656494049824|
| stddev|  9.524509591970393|
|    min|-22.949825936196074|
|    max| 22.647750304177556|
+-------+-------------------+<jupyter_text>Train Linear Regression & Evaluate on Test Set<jupyter_code>better_model = lr.fit(train)
test_results = better_model.evaluate(test)
test_results.residuals.show()
test_results.rootMeanSquaredError
test_results.r2<jupyter_output><empty_output><jupyter_text>Deploying a Model<jupyter_code>unlabeled_data = test.select('features')
unlabeled_data.show()
predictions = better_model.transform(unlabeled_data)
predictions.show()<jupyter_output>+--------------------+--------------------+
|            features|          prediction|
+--------------------+--------------------+
|(10,[0,1,2,3,4,5,...|  4.9147798490833985|
|(10,[0,1,2,3,4,5,...|  1.3045607073775414|
|(10,[0,1,2,3,4,5,...| -3.4711366329188267|
|(10,[0,1,2,3,4,5,...|   -1.34499179254183|
|(10,[0,1,2,3,4,5,...|   -2.10500624408363|
|(10,[0,1,2,3,4,5,...| -0.8316317179511803|
|(10,[0,1,2,3,4,5,...| -2.0145237307862462|
|(10,[0,1,2,3,4,5,...|-0.04229488203451...|
|(10,[0,1,2,3,4,5,...|  3.1305375564044353|
|(10,[0,1,2,3,4,5,...|  3.2048135016176285|
|(10,[0,1,2,3,4,5,...| 0.24587430576960523|
|(10,[0,1,2,3,4,5,...|  2.6936981506222315|
|(10,[0,1,2,3,4,5,...| -2.0405901788495813|
|(10,[0,1,2,3,4,5,...| -0.7749979785075011|
|(10,[0,1,2,3,4,5,...|  1.3237428079774742|
|(10,[0,1,2,3,4,5,...|  -4.025762591448223|
|(10,[0,1,2,3,4,5,...| -1.8940733203536815|
|(10,[0,1,2,3,4,5,...|  2.1217345507722385|
|(10,[0,1,2,3,4,5,...|  4.2537479487371606|
|(10,[0,1,2,3,4,5,...|  3.044746[...]
{'fraction_non_alphanumeric': 0.3171961674230963, 'fraction_numerical': 0.28189611699445283, 'mean_word_length': 3.3604395604395605, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '14853', 'license': 'MIT', 'n_tokens_mistral': 1227, 'n_tokens_neox': 930, 'n_words': 82, 'path': 'apache-spark/python/linear-regression/linear-regression.ipynb', 'repo_name': 'muhdlaziem/notes', 'size': '1983'}