<jupyter_start><jupyter_text>HashingTF and CountVectorizer  Load data<jupyter_code>twitter = spark.createDataFrame([
                                ('Wenqiang is a spark expert', 'Wenqiang', 1.0),
                                ('Ming is learning spark', 'Ming', 0.0)],
                                ['text', 'id', 'label']
                               )
twitter.show()<jupyter_output>+--------------------+--------+-----+
|                text|      id|label|
+--------------------+--------+-----+
|Wenqiang is a spa...|Wenqiang|  1.0|
|Ming is learning ...|    Ming|  0.0|
+--------------------+--------+-----+<jupyter_text>Tokenization<jupyter_code>from pyspark.ml.feature import Tokenizer
tokenizer_mod = Tokenizer(inputCol='text', outputCol='tokens')
twitter_tokens = tokenizer_mod.transform(twitter)
twitter_tokens.show()<jupyter_output>+--------------------+--------+-----+--------------------+
|                text|      id|label|              tokens|
+--------------------+--------+-----+--------------------+
|Wenqiang is a spa...|Wenqiang|  1.0|[wenqiang, is, a,...|
|Ming is learning ...|    Ming|  0.0|[ming, is, learni...|
+--------------------+--------+-----+--------------------+<jupyter_text>HashingTF<jupyter_code>from pyspark.ml.feature import HashingTF
hashingTF_mod = HashingTF(numFeatures=pow(2,4), inputCol='tokens', outputCol='features')
hashingTF_twitter = hashingTF_mod.transform(twitter_tokens)
hashingTF_twitter.show(truncate=False)<jupyter_output>+--------------------------+--------+-----+--------------------------------+---------------------------------+
|text                      |id      |label|tokens                          |features                         |
+--------------------------+--------+-----+--------------------------------+---------------------------------+
|Wenqiang is a spark expert|Wenqiang|1.0  |[wenqiang, is, a, spark, expert]|(16,[1,2,9,13],[2.0,1.0,1.0,1.0])|
|Ming is learning spark    |Ming    |0.0  |[ming, is, learning, spark]     |(16,[0,1,14],[1.0,2.0,1.0])      |
+--------------------------+--------+-----+--------------------------------+---------------------------------+<jupyter_text>CountVectorizer<jupyter_code>from pyspark.ml.feature import CountVectorizer
count_vectorizer = CountVectorizer(vocabSize=pow(2,4), inputCol='tokens', outputCol='features')
countVectorizer_mod = count_vectorizer.fit(twitter_tokens)
countVectorizer_twitter = countVectorizer_mod.transform(twitter_tokens)
countVectorizer_twitter.show(truncate=False)<jupyter_output>+--------------------------+--------+-----+--------------------------------+-------------------------------------+
|text                      |id      |label|tokens                          |features                             |
+--------------------------+--------+-----+--------------------------------+-------------------------------------+
|Wenqiang is a spark expert|Wenqiang|1.0  |[wenqiang, is, a, spark, expert]|(7,[0,1,2,3,5],[1.0,1.0,1.0,1.0,1.0])|
|Ming is learning spark    |Ming    |0.0  |[ming, is, learning, spark]     |(7,[0,1,4,6],[1.0,1.0,1.0,1.0])      |
+--------------------------+--------+-----+--------------------------------+-------------------------------------+
{'fraction_non_alphanumeric': 0.39981331673926573, 'fraction_numerical': 0.024891101431238332, 'mean_word_length': 1.157718120805369, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '378964', 'license': 'MIT', 'n_tokens_mistral': 943, 'n_tokens_neox': 893, 'n_words': 136, 'path': 'notebooks/ipynb/HashingTF-and-CountVectorizer.ipynb', 'repo_name': 'peymanyousefianasml/learning-apache-spark', 'size': '3214'}