<jupyter_start><jupyter_text>Load the data:<jupyter_code>df = spark.read.options(header = True, inferSchema = True).csv("drive/MyDrive/Colab Notebooks/dog_food.csv"); df<jupyter_output><empty_output><jupyter_text>Analyze each column:<jupyter_code>df.printSchema() # No strings as columns values, so we don't need to index it.
df.columns<jupyter_output><empty_output><jupyter_text>Create features column:<jupyter_code>from pyspark.ml.feature import VectorAssembler, StringIndexer
assembler = VectorAssembler(inputCols=['A','B','C','D'], outputCol='features')
final_data = assembler.transform(df); final_data.toPandas()
from pyspark.ml.classification import (RandomForestClassifier)
train, test = final_data.randomSplit([0.7, 0.3], 42)<jupyter_output><empty_output><jupyter_text>Create our classifier:<jupyter_code>rfc = RandomForestClassifier(numTrees = 100, labelCol='Spoiled')<jupyter_output><empty_output><jupyter_text>Train and predict:<jupyter_code>rfc_model = rfc.fit(train)
predictions = rfc_model.transform(test)<jupyter_output><empty_output><jupyter_text>Check the predictions:<jupyter_code>predictions.toPandas()<jupyter_output><empty_output><jupyter_text>Evaluate our classifier:<jupyter_code>from pyspark.ml.evaluation import BinaryClassificationEvaluator
evaluator = BinaryClassificationEvaluator(rawPredictionCol='prediction', labelCol='Spoiled')
evaluator.evaluate(predictions)<jupyter_output><empty_output><jupyter_text>¿Cuál es es el conservante que más efecto tiene?Para ello, empleamos la función featureImportances que forma parte de la función del modelo. Esta función nos muestra la importancia de cada 'feature' respecto a la predicción. Podemos ver, sin ninguna duda, que la columna con index 2 (C) es la que más importancia tiene con muchísima diferencia.Por tanto, podemos concluír que el conservante C es el que más impacto tiene en que se estropee la comida para perros.<jupyter_code>rfc_model.featureImportances<jupyter_output><empty_output><jupyter_text>BONUS TRACK Let's do it with a pipeline.<jupyter_code>#First, we need to import the function
from pyspark.ml import Pipeline<jupyter_output><empty_output><jupyter_text>Create the stages that will follow our pipeline<jupyter_code>pip_assem = VectorAssembler(inputCols=['A', 'B', 'C', 'D'], outputCol= 'features')
pip_rfc = RandomForestClassifier(numTrees= 100, labelCol= 'Spoiled')
pipeline = Pipeline(stages = [pip_assem, pip_rfc])
pip_train, pip_test = df.randomSplit([0.7, 0.3], 42)
model = pipeline.fit(pip_train)
pip_predic = model.transform(pip_test)
pip_predic.toPandas()<jupyter_output><empty_output><jupyter_text>Check the accuracy of our model<jupyter_code>from pyspark.ml.evaluation import BinaryClassificationEvaluator
evaluator = BinaryClassificationEvaluator(rawPredictionCol='prediction', labelCol='Spoiled')
pip_predic = pip_predic.select(['Spoiled', 'prediction'])
evaluator.evaluate(pip_predic)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10533515731874145, 'fraction_numerical': 0.006497948016415869, 'mean_word_length': 9.87360594795539, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '368422', 'license': 'MIT', 'n_tokens_mistral': 1040, 'n_tokens_neox': 1004, 'n_words': 206, 'path': 'Spark/BDE_Task_4.ipynb', 'repo_name': 'biocompt/My_Work_In_The_Master', 'size': '2924'}