<jupyter_start><jupyter_text>Logistic Regression Classification<jupyter_code>from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml import Pipeline
from pyspark.mllib.regression import LabeledPoint
from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import StringIndexer
from pyspark.mllib.evaluation import MulticlassMetrics
from pyspark.ml.tuning import ParamGridBuilder, TrainValidationSplit<jupyter_output><empty_output><jupyter_text>Utility function to create the appropriate data frame for classification algorithms in MLlib<jupyter_code>def mapLibSVM(row): 
    return (row[5],Vectors.dense(row[:3]))<jupyter_output><empty_output><jupyter_text>create the dataframe from a csv<jupyter_code>df = spark.read \
        .format("csv") \
        .option("header", "true") \
        .option("inferSchema", "true") \
        .load("datasets/iris.data")<jupyter_output><empty_output><jupyter_text>Classification algorithms requires numeric values for labels<jupyter_code>indexer = StringIndexer(inputCol="label", outputCol="labelIndex")
indexer = indexer.fit(df).transform(df)
indexer.show()
dfLabeled = indexer.rdd.map(mapLibSVM).toDF(["label", "features"])
dfLabeled.show()
train, test = dfLabeled.randomSplit([0.9, 0.1], seed=12345)<jupyter_output><empty_output><jupyter_text>schema verification<jupyter_code>train.printSchema()<jupyter_output><empty_output><jupyter_text>Instantiate the Logistic Regression and the pipeline.<jupyter_code>lr = LogisticRegression(labelCol="label", maxIter=10)<jupyter_output><empty_output><jupyter_text>We use a *ParamGridBuilder* to construct a grid of parameters to search over.*TrainValidationSplit* will try all combinations of values and determine best model using the evaluator.<jupyter_code>paramGrid = ParamGridBuilder()\
    .addGrid(lr.regParam, [0.1, 0.001]) \
    .build()<jupyter_output><empty_output><jupyter_text>In this case the estimator is simply the linear regression.A TrainValidationSplit requires an Estimator, a set of Estimator ParamMaps, and an Evaluator.<jupyter_code>tvs = TrainValidationSplit(estimator=lr,
                           estimatorParamMaps=paramGrid,
                           evaluator=RegressionEvaluator(),
                           # 80% of the data will be used for training, 20% for validation.
                           trainRatio=0.9)<jupyter_output><empty_output><jupyter_text>Fit the pipeline to training documents.<jupyter_code>model = tvs.fit(train)<jupyter_output><empty_output><jupyter_text>Compute the predictions from the model<jupyter_code>result = model.transform(test)
predictions = result.select(["prediction", "label"])
predictions.show()
# Instantiate metrics object
metrics = MulticlassMetrics(predictions.rdd)

# Overall statistics
print("Summary Stats")
print("Precision = %s" % metrics.precision())
print("Recall = %s" % metrics.recall())
print("F1 Score = %s" % metrics.fMeasure())
print("Accuracy = %s" % metrics.accuracy)
# Weighted stats
print("Weighted recall = %s" % metrics.weightedRecall)
print("Weighted precision = %s" % metrics.weightedPrecision)
print("Weighted F(1) Score = %s" % metrics.weightedFMeasure())
print("Weighted F(0.5) Score = %s" % metrics.weightedFMeasure(beta=0.5))
print("Weighted false positive rate = %s" % metrics.weightedFalsePositiveRate)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11127508854781581, 'fraction_numerical': 0.009149940968122786, 'mean_word_length': 6.00206611570248, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '85794', 'license': 'MIT', 'n_tokens_mistral': 1098, 'n_tokens_neox': 1061, 'n_words': 222, 'path': '2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb', 'repo_name': 'nicolassoares20/data-science-cookbook', 'size': '3388'}