<jupyter_start><jupyter_text>Logistic Regression using spark MLlib The dataset is the Pima Indians Diabetes Dataset [https://www.kaggle.com/uciml/pima-indians-diabetes-database/version/1]. This dataset belongs to National Institute of Diabetes and Digestive and Kidney Diseases. The dataset contains data used to classifiy ifsomeone has diabetes or not. The dataset contains felmale person's data. Features include: number of pregnancies, Glucose concenteration, Blood pressure, skinThickness, Insulin, BMI, DiabetesPedigreeFunction,Age. The label is 1 for diabetic and zero for non diabetic. 268 Participants out of 768 are 1(diabetec).<jupyter_code>from pyspark.sql import SparkSession #import spark session
spark= SparkSession.builder.appName("MLlib demo").getOrCreate() #Create a spark session using MLlib demo as name
#conda install -c conda-forge pyspark<jupyter_output><empty_output><jupyter_text>Load Diabetes Datast stored<jupyter_code>#Load diabetes.csv from loca storage to a data frame using spark, header=True <The first row is column header>, inferSchema=True, 
#option to inferSchema directly from the dataset
diab_ds=spark.read.csv('diabetes.csv', header=True, inferSchema=True)<jupyter_output><empty_output><jupyter_text>Get the name of features of the dataset<jupyter_code>diab_ds.columns<jupyter_output><empty_output><jupyter_text>Get the datatype of the features<jupyter_code>diab_ds.dtypes<jupyter_output><empty_output><jupyter_text>Read First row of dataset<jupyter_code>diab_ds.head() #returns the first row as key and value<jupyter_output><empty_output><jupyter_text>In order to see the number of examples for each class, we use groupBy outcome and apply count operation. The result shows that we have 268 diabetic and 500 non diabetic examples. This shows that this dataset is unbalanced.<jupyter_code>diab_ds.groupBy('Outcome').count().show()<jupyter_output>+-------+-----+
|Outcome|count|
+-------+-----+
|      1|  268|
|      0|  500|
+-------+-----+<jupyter_text>Dataset preparataion and preprocessing for Logistic regression We need Vector Assembler  which is a transformer that combines a set of selected features in to aa single feature vector. For example for our dataset with 8 features, it will combine these 8 features in to one feature vector.<jupyter_code>#!pip install numpy
#!pip install numpy
#import vectors and vector assembler
from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler
#create vectorAssembler transformer that takes all features and maps them in to one vector called <features>
assembler= VectorAssembler(
inputCols=['Pregnancies','Glucose','BloodPressure','SkinThickness','Insulin','BMI','DiabetesPedigreeFunction',
           'Age'],
    outputCol="features"

)
diab_ds_vec=assembler.transform(diab_ds) #Transform diab_db using assembler
diab_ds_vec.show(3) #show the first element<jupyter_output>+-----------+-------+-------------+-------------+-------+----+------------------------+---+-------+--------------------+
|Pregnancies|Glucose|BloodPressure|SkinThickness|Insulin| BMI|DiabetesPedigreeFunction|Age|Outcome|            features|
+-----------+-------+-------------+-------------+-------+----+------------------------+---+-------+--------------------+
|          6|    148|           72|           35|      0|33.6|                   0.627| 50|      1|[6.0,148.0,72.0,3...|
|          1|     85|           66|           29|      0|26.6|                   0.351| 31|      0|[1.0,85.0,66.0,29...|
|          8|    183|           64|            0|      0|23.3|                   0.672| 32|      1|[8.0,183.0,64.0,0...|
+-----------+-------+-------------+-------------+-------+----+------------------------+---+-------+--------------------+
only showing top 3 rows<jupyter_text>Now we select features and outcome to build our dataset ready for preprocessing<jupyter_code>diab_db_final= diab_ds_vec.select('features','Outcome') #select features and Outcome from diab_ds_vec
diab_db_final.show(3) #show first three elements<jupyter_output>+--------------------+-------+
|            features|Outcome|
+--------------------+-------+
|[6.0,148.0,72.0,3...|      1|
|[1.0,85.0,66.0,29...|      0|
|[8.0,183.0,64.0,0...|      1|
+--------------------+-------+
only showing top 3 rows<jupyter_text>Feature scalingSince the values of each feture are in different scale, we apply scaling to put all featues on the same scale. We use Standard Scaler which is MLlib transformer that transformes a dataset of vector rows by scaling each feature to have a zero mean or unit standard devation. Feature scaling can imporve accuracy for some classifiers<jupyter_code>from pyspark.ml.feature import StandardScaler
#withStd converts features to unit standard deviation and withMean: centers the data withMean before scaling
scaler=StandardScaler(inputCol="features", outputCol="scaledFeatures", withStd=True, withMean=False)
#Now build scaler model using diab_db_final
ScalerModel=scaler.fit(diab_db_final)
#Transform diab_db_final using the scaler model
diab_db_scaled= ScalerModel.transform(diab_db_final)
diab_db_scaled.show(1) #show first row of scaled dataset<jupyter_output>+--------------------+-------+--------------------+
|            features|Outcome|      scaledFeatures|
+--------------------+-------+--------------------+
|[6.0,148.0,72.0,3...|      1|[1.78063837321943...|
+--------------------+-------+--------------------+
only showing top 1 row<jupyter_text>Now we split our scaled dataset in to training and testing data. We use 75% for training and 25% for testing<jupyter_code>diab_db_train, diab_db_test=diab_db_scaled.select('scaledFeatures','Outcome').randomSplit([0.75,0.25])
diab_db_train.show(3)<jupyter_output>+--------------------+-------+
|      scaledFeatures|Outcome|
+--------------------+-------+
|(8,[0,1,6,7],[0.5...|      0|
|(8,[0,1,6,7],[0.8...|      0|
|(8,[0,1,6,7],[1.7...|      0|
+--------------------+-------+
only showing top 3 rows<jupyter_text>Logistic Regression classiferLogistic regression is used to solve binary classification problem. Binomial logistic regeression is used for binary classification and multnomial is used for multi-class classification<jupyter_code>from pyspark.ml.classification import LogisticRegression
lrModel= LogisticRegression(maxIter=50, featuresCol='scaledFeatures', labelCol='Outcome') #build a model by specifying the labelCol as Outcome
lrModel=lrModel.fit(diab_db_train) # Train the model using diab_db_train
trainingSummary= lrModel.summary<jupyter_output><empty_output><jupyter_text>Evaluation of logistic regression using test data<jupyter_code>from pyspark.ml.evaluation import BinaryClassificationEvaluator #We use this to evaluate all binary classifications
predictions= lrModel.evaluate(diab_db_test) #apply classification on test data
predictions.predictions.show()
#evaluator= BinaryClassificationEvaluator(rawPredictionCol='predection', labelCol="Outcome")
#evaluator.evaluate(predections.predection)


exit()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1846891301282955, 'fraction_numerical': 0.030029606654448048, 'mean_word_length': 3.1485380116959063, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '664891', 'license': 'MIT', 'n_tokens_mistral': 2255, 'n_tokens_neox': 2078, 'n_words': 592, 'path': 'py/SparkMachineLearningExample.ipynb', 'repo_name': 'roveicy/tutorial-spark-cluster', 'size': '7093'}