<jupyter_start><jupyter_text>Spark-NLP The DataThe UCI ML News Aggregator dataset contains headlines and categories for over 400k news articles. Task is to build classification model that classifies news headline into pre-defined categories.The data can be downloaded from https://www.kaggle.com/uciml/news-aggregator-dataset  Prerequisites:Basics understand of Pythonhttps://www.kaggle.com/learn/pythonNatural Language Processing  https://www.analyticsvidhya.com/blog/2017/01/ultimate-guide-to-understand-implement-natural-language-processing-codes-in-python/Apache Sparkhttps://docs.databricks.com/spark/latest/gentle-introduction/gentle-intro.htmlhttps://docs.databricks.com/spark/latest/gentle-introduction/gentle-intro.htmlgentle-introduction-to-apache-sparkhttps://docs.databricks.com/spark/latest/gentle-introduction/for-data-scientists.html  Importing the necessary libraries<jupyter_code>from pyspark.ml import Pipeline 
from pyspark.ml.feature import CountVectorizer,StringIndexer, RegexTokenizer,StopWordsRemover
from pyspark.sql.functions import col, udf,regexp_replace,isnull
from pyspark.sql.types import StringType,IntegerType
from pyspark.ml.classification import NaiveBayes
from pyspark.ml.evaluation import MulticlassClassificationEvaluator<jupyter_output><empty_output><jupyter_text>Data Ingestion and Extraction In this example, I am importing dataset from S3(https://docs.databricks.com/spark/latest/data-sources/aws/amazon-s3.html).You can can directly upload dataset to databricks cloud(https://docs.databricks.com/user-guide/importing-data.htmlimport-data)<jupyter_code>s3_bucket_path = "/mnt/lp-dataset/news-aggregator/uci-news-aggregator.csv"
news_data = spark.read.csv(s3_bucket_path,header = 'True',inferSchema='True')
news_data.show()
news_data.count()<jupyter_output><empty_output><jupyter_text>There are 422937 news articles d Select the columns needed for analysis<jupyter_code>title_category = news_data.select("TITLE","CATEGORY")
title_category.show()<jupyter_output><empty_output><jupyter_text>Let's check null values in TITLE and CATEGORY columns<jupyter_code>def null_value_count(df):
  null_columns_counts = []
  numRows = df.count()
  for k in df.columns:
    nullRows = df.where(col(k).isNull()).count()
    if(nullRows > 0):
      temp = k,nullRows
      null_columns_counts.append(temp)
  return(null_columns_counts)
null_columns_count_list = null_value_count(title_category)

spark.createDataFrame(null_columns_count_list, ['Column_With_Null_Value', 'Null_Values_Count']).show()<jupyter_output><empty_output><jupyter_text>There are 389 empty titles and 516 categories Drop the null values<jupyter_code>title_category = title_category.dropna()
title_category.count()
title_category.show(truncate=False)
title_category.select("Category").distinct().count()<jupyter_output><empty_output><jupyter_text>Top 20 news categories<jupyter_code>title_category.groupBy("Category").count().orderBy(col("count").desc()).show(truncate=False)<jupyter_output><empty_output><jupyter_text>Top 20 news title<jupyter_code>title_category.groupBy("TITLE").count().orderBy(col("count").desc()).show(truncate=False)<jupyter_output><empty_output><jupyter_text>Let's clean the dataset Removing numbers from titles<jupyter_code>title_category = title_category.withColumn("only_str",regexp_replace(col('TITLE'), '\d+', ''))
title_category.select("TITLE","only_str").show(truncate=False)<jupyter_output><empty_output><jupyter_text>Spliting text into words<jupyter_code>regex_tokenizer = RegexTokenizer(inputCol="only_str", outputCol="words", pattern="\\W")
raw_words = regex_tokenizer.transform(title_category)
raw_words.show()<jupyter_output><empty_output><jupyter_text>Removing stop words<jupyter_code>remover = StopWordsRemover(inputCol="words", outputCol="filtered")
words_df = remover.transform(raw_words)
words_df.select("words","filtered").show(truncate=False)<jupyter_output><empty_output><jupyter_text>Lets encode column of category to a column of category indices<jupyter_code>indexer = StringIndexer(inputCol="CATEGORY", outputCol="categoryIndex")
feature_data = indexer.fit(words_df).transform(words_df)
feature_data.select("CATEGORY","categoryIndex").show()<jupyter_output><empty_output><jupyter_text>Converting text into vectors of token counts.<jupyter_code>cv = CountVectorizer(inputCol="filtered", outputCol="features")
model = cv.fit(words_label_index)
countVectorizer_feateures = model.transform(words_label_index)<jupyter_output><empty_output><jupyter_text>Partition Training & Test sets<jupyter_code>(trainingData, testData) = countVectorizer_feateures.randomSplit([0.8, 0.2],seed = 11)<jupyter_output><empty_output><jupyter_text>Model Training and Evaluation<jupyter_code>nb = NaiveBayes(modelType="multinomial",labelCol="categoryIndex", featuresCol="features")
nbModel = nb.fit(trainingData)
nb_predictions = nbModel.transform(testData)
nb_predictions.select("prediction", "categoryIndex", "features").show(5)
evaluator = MulticlassClassificationEvaluator(labelCol="categoryIndex", predictionCol="prediction", metricName="accuracy")
nb_accuracy = evaluator.evaluate(nb_predictions)
print("Accuracy of NaiveBayes is = %g"% (nb_accuracy))
print("Test Error of NaiveBayes = %g " % (1.0 - nb_accuracy))<jupyter_output><empty_output><jupyter_text>Reference https://spark.apache.org/docs/latest/ml-guide.html<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11947231512448904, 'fraction_numerical': 0.007246376811594203, 'mean_word_length': 9.596456692913385, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '260386', 'license': 'MIT', 'n_tokens_mistral': 1838, 'n_tokens_neox': 1742, 'n_words': 254, 'path': 'pyspark-nlp-kaggle.ipynb', 'repo_name': 'garbamoussa/Spark_notebook', 'size': '5382'}