<jupyter_start><jupyter_text>Logistic Classification<jupyter_code>from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('classification').getOrCreate()
data = spark.read.csv('customer_churn.csv',inferSchema=True,header=True)
data.printSchema()
data.show()<jupyter_output>+-------------------+----+--------------+---------------+-----+---------+-------------------+--------------------+--------------------+-----+
|              Names| Age|Total_Purchase|Account_Manager|Years|Num_Sites|       Onboard_date|            Location|             Company|Churn|
+-------------------+----+--------------+---------------+-----+---------+-------------------+--------------------+--------------------+-----+
|   <NAME>|42.0|       11066.8|              0| 7.22|      8.0|2013-08-30 07:00:40|10265 Elizabeth M...|          Harvey LLC|    1|
|      <NAME>|41.0|      11916.22|              0|  6.5|     11.0|2013-08-13 00:38:46|6157 Frank Garden...|          Wilson PLC|    1|
|        <NAME>|38.0|      12884.75|              0| 6.67|     12.0|2016-06-29 06:20:07|1331 Keith Court ...|Miller, Johnson a...|    1|
|      <NAME>|42.0|       8010.76|              0| 6.71|     10.0|2014-04-22 12:43:12|13120 Daniel Moun...|           Smith Inc|    1|
|     [...]<jupyter_text>Check out the data<jupyter_code>data.describe().show()
data.columns
from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=[
 'Age',
 'Total_Purchase',
 'Account_Manager',
 'Years',
 'Num_Sites'
],outputCol='features')
output = assembler.transform(data)
final_data = output.select('features','churn')<jupyter_output><empty_output><jupyter_text>Test Train Split<jupyter_code>train_churn,test_churn = final_data.randomSplit([0.7,0.3])<jupyter_output><empty_output><jupyter_text>Fit the model<jupyter_code>from pyspark.ml.classification import LogisticRegression
lr_churn = LogisticRegression(labelCol='churn')
fitted_churn_model = lr_churn.fit(train_churn)
training_sum = fitted_churn_model.summary
training_sum.predictions.describe().show()<jupyter_output>+-------+-------------------+-------------------+
|summary|              churn|         prediction|
+-------+-------------------+-------------------+
|  count|                629|                629|
|   mean| 0.1685214626391097|0.12559618441971382|
| stddev|0.37462661870173564|  0.331657441239251|
|    min|                0.0|                0.0|
|    max|                1.0|                1.0|
+-------+-------------------+-------------------+<jupyter_text>Evaluate resultsLet's evaluate the results on the data set we were given (using the test data)<jupyter_code>from pyspark.ml.evaluation import BinaryClassificationEvaluator
pred_and_labels = fitted_churn_model.evaluate(test_churn)
pred_and_labels.predictions.show()<jupyter_output>+--------------------+-----+--------------------+--------------------+----------+
|            features|churn|       rawPrediction|         probability|prediction|
+--------------------+-----+--------------------+--------------------+----------+
|[25.0,9672.03,0.0...|    0|[4.72066593809958...|[0.99116943013004...|       0.0|
|[28.0,9090.43,1.0...|    0|[1.55698501872663...|[0.82592029702177...|       0.0|
|[28.0,11128.95,1....|    0|[4.29930929518062...|[0.98660395647775...|       0.0|
|[28.0,11245.38,0....|    0|[3.76613733174476...|[0.97738212849841...|       0.0|
|[29.0,5900.78,1.0...|    0|[4.13573220108600...|[0.98426073360767...|       0.0|
|[29.0,10203.18,1....|    0|[3.85488639799717...|[0.97926311563551...|       0.0|
|[30.0,6744.87,0.0...|    0|[3.47949087979864...|[0.97009855550350...|       0.0|
|[30.0,8677.28,1.0...|    0|[4.18660567655569...|[0.98502973128938...|       0.0|
|[30.0,10960.52,1....|    0|[2.46736693453430...|[0.92182222097137...|       0.0|
|[30.0,13473.35,[...]<jupyter_text>Using AUC<jupyter_code>churn_eval = BinaryClassificationEvaluator(rawPredictionCol='prediction',
                                           labelCol='churn')
auc = churn_eval.evaluate(pred_and_labels.predictions)
auc<jupyter_output><empty_output><jupyter_text>Predict on brand new unlabeled dataWe still need to evaluate the new_customers.csv file!<jupyter_code>final_lr_model = lr_churn.fit(final_data)
new_customers = spark.read.csv('new_customers.csv',inferSchema=True,
                              header=True)
new_customers.printSchema()
test_new_customers = assembler.transform(new_customers)
test_new_customers.printSchema()
final_results = final_lr_model.transform(test_new_customers)
final_results.select('Company','prediction').show()<jupyter_output>+----------------+----------+
|         Company|prediction|
+----------------+----------+
|        King Ltd|       0.0|
|   Cannon-Benson|       1.0|
|Barron-Robertson|       1.0|
|   Sexton-Golden|       1.0|
|        Wood LLC|       0.0|
|   Parks-Robbins|       1.0|
+----------------+----------+
{'fraction_non_alphanumeric': 0.2733224222585925, 'fraction_numerical': 0.13236497545008183, 'mean_word_length': 1.9066587395957193, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '74966', 'license': 'Apache-2.0', 'n_tokens_mistral': 2023, 'n_tokens_neox': 1690, 'n_words': 210, 'path': '5-ML/Classification.ipynb', 'repo_name': 'PrathmeshRahate/30-days-of-pyspark', 'size': '4917'}