<jupyter_start><jupyter_text>Install required libraries<jupyter_code>! pip install hazm
! pip install gensim
! pip -q install clean-text[gpl]
! pip install pandas
! pip install arabic_reshaper
! pip install python-bidi
! pip install seaborn
! pip install parsivar
! pip install stanza
!pip install -q findspark
!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip ngrok-stable-linux-amd64.zip
get_ipython().system_raw('./ngrok http 4050 &')<jupyter_output>--2021-12-13 18:48:46--  https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
Resolving bin.equinox.io (bin.equinox.io)... 172.16.17.32, 192.168.127.12, 192.168.127.12, ...
Connecting to bin.equinox.io (bin.equinox.io)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13832437 (13M) [application/octet-stream]
Saving to: ‘ngrok-stable-linux-amd64.zip.2’

ngrok-stable-linux- 100%[===================>]  13.19M  54.1MB/s    in 0.2s    

2021-12-13 18:48:46 (54.1 MB/s) - ‘ngrok-stable-linux-amd64.zip.2’ saved [13832437/13832437]

Archive:  ngrok-stable-linux-amd64.zip
replace ngrok? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
  inflating: ngrok<jupyter_text>Install Open JDK for Spark and Hadoop<jupyter_code>!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://dlcdn.apache.org/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.2.tgz
!tar xf spark-3.2.0-bin-hadoop3.2.tgz<jupyter_output><empty_output><jupyter_text>Download Masnavi corpus to work on<jupyter_code>!wget https://raw.githubusercontent.com/SajjadMb/Masnavi_NLP_Toolkit/main/masnavi.txt?token=<KEY>
!mv masnavi.txt?token=AF4HC2VV75CFQKTTASY7GXDBX5N2Y masnavi.txt<jupyter_output>--2021-12-13 18:49:56--  https://raw.githubusercontent.com/SajjadMb/Masnavi_NLP_Toolkit/main/masnavi.txt?token=<KEY>
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.3.11, 192.168.3.11, 172.16.17.32, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3090459 (2.9M) [text/plain]
Saving to: ‘masnavi.txt?token=<KEY>’

masnavi.txt?token=A 100%[===================>]   2.95M  --.-KB/s    in 0.07s   

2021-12-13 18:49:57 (40.4 MB/s) - ‘masnavi.txt?token=<KEY>’ saved [3090459/3090459]<jupyter_text>Download Stopwords (I added some necessary stop words after studying the corpus)<jupyter_code>!wget https://raw.githubusercontent.com/SajjadMb/Masnavi_NLP_Toolkit/main/stopwords.txt?token=<KEY>
!mv stopwords.txt?token=AF4HC2UWVYPTWNSXDWDW6RTBX5OMU stopwords.txt<jupyter_output>--2021-12-13 18:49:57--  https://raw.githubusercontent.com/SajjadMb/Masnavi_NLP_Toolkit/main/stopwords.txt?token=<KEY>
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.3.11, 192.168.3.11, 172.16.17.32, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15569 (15K) [text/plain]
Saving to: ‘stopwords.txt?token=<KEY>’


          stopwords   0%[                    ]       0  --.-KB/s               
stopwords.txt?token 100%[===================>]  15.20K  --.-KB/s    in 0.001s  

2021-12-13 18:49:57 (27.6 MB/s) - ‘stopwords.txt?token=<KEY>’ saved [15569/15569]<jupyter_text>Import Libraries<jupyter_code>import pandas as pd
import numpy as np
import re
from __future__ import unicode_literals
import hazm
import nltk
import codecs
import tqdm
import gensim
from cleantext import clean<jupyter_output><empty_output><jupyter_text>Plot Utilsfollowing function is responsible for plotting top 8 common categories resulting from LDA and NMF algorithms<jupyter_code>def plot_top_words(model, feature_names, n_top_words, title):
    fig, axes = plt.subplots(2, 5, figsize=(30, 15), sharex=True)
    axes = axes.flatten()
    for topic_idx, topic in enumerate(model.components_):
        top_features_ind = topic.argsort()[: -n_top_words - 1 : -1]
        top_features = [feature_names[i] for i in top_features_ind]
        weights = topic[top_features_ind]

        ax = axes[topic_idx]
        ax.barh(top_features, weights, height=0.7)
        ax.set_title(f"Topic {topic_idx +1}", fontdict={"fontsize": 30})
        ax.invert_yaxis()
        ax.tick_params(axis="both", which="major", labelsize=20)
        for i in "top right left".split():
            ax.spines[i].set_visible(False)
        fig.suptitle(title, fontsize=40)

    plt.subplots_adjust(top=0.90, bottom=0.05, wspace=0.90, hspace=0.3)
    plt.show()<jupyter_output><empty_output><jupyter_text>Label Persian Compatibilityfollowing functions are needed to print persian labels on plot in right to left direction<jupyter_code>from arabic_reshaper import reshape
from bidi.algorithm import get_display
import seaborn as sns
def make_farsi_text(x):
    if x == 'الله':
        return 'هللا'
    reshaped_text = reshape(x)
    farsi_text = get_display(reshaped_text)
    return farsi_text<jupyter_output><empty_output><jupyter_text>Load DataSet from `masnavi.txt`<jupyter_code>with open('masnavi.txt', 'r', encoding="utf8") as infile:
    masnavi_file = infile.readlines()<jupyter_output><empty_output><jupyter_text>Load Stopwords from `stopwords.txt`<jupyter_code>stopwords = [x.strip() for x in codecs.open('stopwords.txt','r','utf-8').readlines()]<jupyter_output><empty_output><jupyter_text>Instantiate Hazm Normalizer into `normalizer` variable<jupyter_code>normalizer = hazm.Normalizer()
normalizer.normalize('آتشی عشق کاندر نی فتاد')<jupyter_output><empty_output><jupyter_text>Instantiate Hazm Lemmatizer into `lemmatizer` variable<jupyter_code>lemmatizer = hazm.Lemmatizer()
lemmatizer.lemmatize('آتشی عشق کاندر نی فتاد')<jupyter_output><empty_output><jupyter_text>Remove Space patterns regexThe following function returns regex appropriate to omit unwanted spaces in text.<jupyter_code>def remove_spaces_reg():
    # return space patterns
    space_pattern = r"[\xad\ufeff\u200e\u200d\u200b\x7f\u202a\u2003\xa0\u206e\u200c\x9d]"
    return re.compile(space_pattern)<jupyter_output><empty_output><jupyter_text>Remove Symbols and Punctuation patterns regexThe following function returns regex appropriate to omit unwanted symbols and punctuations in text.<jupyter_code>def remove_symbols_reg():
    # return symbol patterns
    symbol_patterns = r"(\d|\"|'ٍ|¬|[؛“،,”‘۔’’‘–]|[|\.÷+\:\-\?»\=\{}\*«»_…\؟!/ـ]|[۰'ٓ۫'ٔ]|[ٓٔ]|[ًٌٍْﹼ،َُِّ«ٰ»ٖء])"
    return re.compile(symbol_patterns)<jupyter_output><empty_output><jupyter_text>Convert Arabic alphabet to persian alphabetThe following function converts the Arabic alphabet in text into the corresponding Persian one.<jupyter_code>def remeove_arabic(text):
    # remove arabic alphabet
    mapping = {
        u"ۀ" : u"ه",
        u"ة" : u"ت",
        u"ي" : u"ی",
        u"ؤ" : u"و",
        u"إ" : u"ا",
        u"ٹ" : u"ت",
        u"ڈ" : u"د",
        u"ئ" : u"ی",
        u"ﻨ" : u"ن",
        u"ﺠ" : u"ج",
        u"ﻣ" : u"م",
        u"ﷲ" : u"",
        u"ﻳ" : u"ی",
        u"ٻ" : u"ب",
        u"ٱ" : u"ا",
        u"ڵ" : u"ل",
        u"ﭘ" : u"پ",
        u"ﻪ" : u"ه",
        u"ﻳ" : u"ی",
        u"ٻ" : u"ب",
        u"ں" : u"ن",
        u"ٶ" : u"و",
        u"ٲ" : u"ا",
        u"ہ" : u"ه",
        u"ﻩ" : u"ه",
        u"ﻩ" : u"ه",
        u"ك" : u"ک",
        u"ﺆ" : u"و",
        u"أ" : u"ا",
        u"ﺪ" : u"د"
    }

    arabic_keys =  re.compile(r"(" + "|".join(mapping.keys()) + r")")
    return arabic_keys.sub(lambda x: mapping[x.group()], text)<jupyter_output><empty_output><jupyter_text>Create needed regexes<jupyter_code>spaces_reg = remove_spaces_reg() 
symbols_reg = remove_symbols_reg()<jupyter_output><empty_output><jupyter_text>clean_text functionthe following method cleans input text from extra spaces, stop words, punctuations, unwanted symbols and arabic alphabet.<jupyter_code>def clean_text(text):
    text = re.sub(spaces_reg, " ", text)
    text = re.sub(symbols_reg, "", text)
    text = remeove_arabic(text)
    text = normalizer.normalize(text)
    regex = r"\b(?:" + "|".join(map(re.escape, stopwords)) + r")\b"
    text = re.sub(regex, " ", text)
    text = re.sub(spaces_reg, " ", text)
    text = re.sub("(\s)+", " ", text)
    text = text.strip()
    return text<jupyter_output><empty_output><jupyter_text>Process Couplet(Beyt)The following code :- checks whether the line is Couplet or not- if the line is couplet extract Poem Number and Couplet number- Extract Hemistich(Mesra)<jupyter_code>def process_couplet(text):
    result = re.search("(\d{1,3})\.(\d{1,3})", text)
    # check if line contains a couplet
    if result:
        pno, cno = result.groups()
        # delete Daftar and Poem number
        couplet = re.sub("(\d{1,3})\.(\d{1,3})", "", text)
        # extract mesra
        hemistich = couplet.split("\t")[1:3]
        # extract important tokens in mesra
        cleaned_hemistich = [clean_text(h) for h in hemistich]
        # clean spaces
        couplet = re.sub(spaces_reg, " ", couplet)
        return pno, cno, couplet, " ".join(cleaned_hemistich), cleaned_hemistich[0], cleaned_hemistich[1]
    return None<jupyter_output><empty_output><jupyter_text>Example 1<jupyter_code>process_couplet('1.10	آتش ِعشق است كاندر نی فتاد	جوشش عشق است كاندر می   فتاد')<jupyter_output><empty_output><jupyter_text>Example 2<jupyter_code>process_couplet('28.16	قطره ای از باده های آسمان	پُر كند جان را ز می و ز ساقیان ')<jupyter_output><empty_output><jupyter_text>Example 3<jupyter_code>process_couplet('10.14	گر نبودی كارش الهام اله	او سگی بودی دراننده، نه شاه')<jupyter_output><empty_output><jupyter_text>Array Masnavi Contains all couplets process by function `process_couplet`<jupyter_code>masnavi = []<jupyter_output><empty_output><jupyter_text>Process Dataset and append all couplets to masnavi list<jupyter_code>daftar = 0
for couplet in masnavi_file:
    if re.search("^(?:دفتر).*.(?:مثنوی)$", couplet):
        daftar += 1
        if daftar == 7:
            print("Processing Completed!")
            break
        print(f"Processing Daftar {daftar}")
    else:
        process_result = process_couplet(couplet)
        if process_result:
            pno = process_result[0]
            cno = process_result[1]
            c = process_result[2]
            cc = process_result[3]
            h1 = process_result[4]
            h2 = process_result[5]
            data = (daftar, pno, cno, c, cc, h1, h2)
            masnavi.append(data)<jupyter_output>Processing Daftar 1
Processing Daftar 2
Processing Daftar 3
Processing Daftar 4
Processing Daftar 5
Processing Daftar 6<jupyter_text>Convert `masnavi` list to `masnavi_df` pandas dataframe<jupyter_code>masnavi_df = pd.DataFrame(masnavi, columns=['Daftar', 'Poem', 'CNo', 'Couplet', 'CCouplet', 'Hemistich1', 'Hemistich2'])<jupyter_output><empty_output><jupyter_text>Convert dataframe Daftar Poem and Couplet Number to Numeric<jupyter_code>masnavi_df['Daftar'] = masnavi_df['Daftar'].apply(pd.to_numeric)
masnavi_df['Poem'] = masnavi_df['Poem'].apply(pd.to_numeric)
masnavi_df['CNo'] = masnavi_df['CNo'].apply(pd.to_numeric)<jupyter_output><empty_output><jupyter_text>Create Tokenized columns<jupyter_code>masnavi_df['Couplet_tokenized'] = masnavi_df['CCouplet'].apply(lambda x:hazm.word_tokenize(x))
masnavi_df['Hemistich1_tokenized'] =  masnavi_df['Hemistich1'].apply(lambda x:hazm.word_tokenize(x))
masnavi_df['Hemistich2_tokenized'] =  masnavi_df['Hemistich2'].apply(lambda x:hazm.word_tokenize(x))<jupyter_output><empty_output><jupyter_text>Pos Tagger  First I tried parsivar with stanford tagger model<jupyter_code>from parsivar import Tokenizer, FindStems, FindChunks, POSTagger

pars_tokenizer = Tokenizer()
pars_tagger = POSTagger(tagging_model="stanford")<jupyter_output><empty_output><jupyter_text>create Pos Tag with Parsivar Tagger<jupyter_code>def create_pos_tag(text):
    # pos tag couplet
    return pars_tagger.parse(pars_tokenizer.tokenize_words(text))<jupyter_output><empty_output><jupyter_text>As you see Results aren't satisfyingMoreover, It's too slow and cannot be apply on whole dataset.  Example 1)<jupyter_code>create_pos_tag('نبودی کارش الهام اله سگی دراننده شاه')<jupyter_output><empty_output><jupyter_text>Example 2)<jupyter_code>create_pos_tag('آتش عشق نی فتاد جوشش عشق فتاد')<jupyter_output><empty_output><jupyter_text>Welcome to SparkI saw the process of post tagging is too time-consuming and the python wrapper implemented by Parsivar is not appropriate to feed into user-defined function because it uses pointers that cannot be pickled at spark runtime process.  First We need to fix environment variables<jupyter_code>import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.2.0-bin-hadoop3.2"<jupyter_output><empty_output><jupyter_text>Then we use `Findspark` to find spark executable<jupyter_code>import findspark
findspark.init()
findspark.find()<jupyter_output><empty_output><jupyter_text>Create Spark Session<jupyter_code>from pyspark.sql import SparkSession

spark = SparkSession.builder\
        .master("local")\
        .appName("Colab")\
        .config('spark.ui.port', '4050')\
        .getOrCreate()<jupyter_output><empty_output><jupyter_text>Import Spark Context and import other necessary utils<jupyter_code>from pyspark.sql import SQLContext
from pyspark import SparkContext
from pyspark.sql.functions import udf, col<jupyter_output><empty_output><jupyter_text>Instanciate `sqlContext` and create spark dataframe from masnavi pandas dataframe<jupyter_code>sqlContext = SQLContext(spark)
masnavi_pdf = sqlContext.createDataFrame(masnavi_df)<jupyter_output>/content/spark-3.2.0-bin-hadoop3.2/python/pyspark/sql/context.py:79: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  FutureWarning<jupyter_text>Print Spark Dataframe Schema<jupyter_code>masnavi_pdf.printSchema()<jupyter_output>root
 |-- Daftar: long (nullable = true)
 |-- Poem: long (nullable = true)
 |-- CNo: long (nullable = true)
 |-- Couplet: string (nullable = true)
 |-- CCouplet: string (nullable = true)
 |-- Hemistich1: string (nullable = true)
 |-- Hemistich2: string (nullable = true)
 |-- Couplet_tokenized: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- Hemistich1_tokenized: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- Hemistich2_tokenized: array (nullable = true)
 |    |-- element: string (containsNull = true)<jupyter_text>Tokenization Using Parsivar and Spark<jupyter_code>from pyspark.sql.types import ArrayType, StringType
token_udf = udf(
    lambda couplet: pars_tokenizer.tokenize_words(couplet),
    StringType())  
 masnavi_pdf = masnavi_pdf.withColumn("ParsTokens", token_udf(masnavi_pdf.Couplet))<jupyter_output><empty_output><jupyter_text>Convert pyspark dataframe to pandas dataframe<jupyter_code>masnavi_df = masnavi_pdf.toPandas()<jupyter_output><empty_output><jupyter_text>store result into `masnavi_df.csv` file<jupyter_code>masnavi_df.to_csv('masnavi_df.csv', encoding='utf-8')<jupyter_output><empty_output><jupyter_text>Stanza Library and Spark  Download persian model<jupyter_code>import stanza
stanza.download('fa')<jupyter_output><empty_output><jupyter_text>Create Processing Pipline<jupyter_code>nlp = stanza.Pipeline(lang='fa', processors='tokenize,mwt,pos')<jupyter_output>2021-12-13 18:52:03 INFO: Loading these models for language: fa (Persian):
=======================
| Processor | Package |
-----------------------
| tokenize  | perdt   |
| mwt       | perdt   |
| pos       | perdt   |
=======================

2021-12-13 18:52:03 INFO: Use device: cpu
2021-12-13 18:52:03 INFO: Loading: tokenize
2021-12-13 18:52:03 INFO: Loading: mwt
2021-12-13 18:52:03 INFO: Loading: pos
2021-12-13 18:52:04 INFO: Done loading processors!<jupyter_text>Execute Stanford model on Masnavi Data using PysparkI reduced the needed time to parse masnavi corpus from 10 hours to 20 minutes; however, if your want to run it yourself, you need to wait for 20 minutes too.  Extract Pos tags without cleaning<jupyter_code>pos_tags_without_cleaning = masnavi_pdf.select("Couplet").rdd.map(lambda x: nlp(x[0])).collect()<jupyter_output><empty_output><jupyter_text>Extract Pos tags with custom cleaning<jupyter_code>pos_tags_with_cleaning = masnavi_pdf.select("Couplet").rdd.map(lambda x: nlp(clean_text(x[0]))).collect()<jupyter_output><empty_output><jupyter_text>Number of pos sentences<jupyter_code>len(pos_tags_with_cleaning)<jupyter_output><empty_output><jupyter_text>PosTag Classthe following class keeps tags data about each word in corpus<jupyter_code>class PosTag:
  def __init__(self, id, text, upos, xpos, feats, start_char, end_char):
    self.id = id
    self.text = text
    self.upos = upos
    self.xpos = xpos
    self.feats = {x.split("=")[0]:x.split("=")[1] for x in [x for x in feats.split("|")]} if feats else {}
    self.start_char = start_char
    self.end_char = end_char

  def to_dict(self):
    return {
        'id': self.id,
        'text': self.text,
        'upos': self.upos,
        'xpos': self.xpos,
        'feats': self.feats,
        'start_char': self.start_char,
        'end_char': self.end_char
    }<jupyter_output><empty_output><jupyter_text>Create `masnavi_tags` containing `PosTag` class objects<jupyter_code>class MasnaviPosBasedAnalyze:
  def __init__(self, pos_tags):
    self.masnavi_tags = list()
    for i, tags in enumerate(pos_tags):
      try:
        sent = tags.sentences[0]
        for word in sent.words:
          self.masnavi_tags.append(PosTag(
              i,
              word.text, 
              word.upos, 
              word.xpos, 
              word.feats, 
              word.start_char, 
              word.end_char))
      except:
        pass
    self.masnavi_pos_df = pd.DataFrame.from_records([t.to_dict() for t in self.masnavi_tags])

  def common_adjectives(self, count=50):
    return nltk.FreqDist(self.masnavi_pos_df[self.masnavi_pos_df["upos"]=="ADJ"]["text"]).most_common(count)

  def common_nouns(self, count=50):
    return nltk.FreqDist(self.masnavi_pos_df[self.masnavi_pos_df["upos"]=="NOUN"]["text"]).most_common(50)

  def common_verbs(self, count=50):
    return nltk.FreqDist(self.masnavi_pos_df[self.masnavi_pos_df["upos"]=="VERB"]["text"]).most_common(50)

  def common_SUBR(self):
    return nltk.FreqDist(uncleaned_analyzer.masnavi_pos_df[uncleaned_analyzer.masnavi_pos_df["xpos"]=='SUBR']['text']).most_common(9)
analyzer = MasnaviPosBasedAnalyze(pos_tags=pos_tags_with_cleaning)<jupyter_output><empty_output><jupyter_text>Most Common Adjectives in **Masnavi**<jupyter_code>analyzer.common_adjectives()<jupyter_output><empty_output><jupyter_text>Most Common Nouns in **Masnavi**<jupyter_code>analyzer.common_nouns()<jupyter_output><empty_output><jupyter_text>Most Common Verbs in **Masnavi**<jupyter_code>analyzer.common_verbs()<jupyter_output><empty_output><jupyter_text>Initial class with uncleaned pos tagging<jupyter_code>uncleaned_analyzer = MasnaviPosBasedAnalyze(pos_tags=pos_tags_without_cleaning)<jupyter_output><empty_output><jupyter_text>Get common SUBR<jupyter_code>uncleaned_analyzer.common_SUBR()<jupyter_output><empty_output><jupyter_text>Frequency Analysis<jupyter_code>from itertools import chain
from collections import Counter<jupyter_output><empty_output><jupyter_text>Extract all words from first tokenizer<jupyter_code>all_words = list(chain.from_iterable(masnavi_df.Couplet_tokenized))
words_frequencies = nltk.FreqDist(all_words).most_common(100)
words_frequencies[:10]
print ('%-16s' % 'Number of words', '%-16s' % len(all_words))
print ('%-16s' % 'Number of unique words', '%-16s' % len(set(all_words)))
avg=np.sum([len(word) for word in all_words])/len(all_words)
print ('%-16s' % 'Average word length', '%-16s' % avg)
print ('%-16s' % 'Longest word', '%-16s' % all_words[np.argmax([len(word) for word in all_words])])<jupyter_output>Number of words  162712          
Number of unique words 22055           
Average word length 4.006084369929692
Longest word     استخوانهاشان<jupyter_text>Extract all words from Stanza tokenized pipeline<jupyter_code>stanza_words = list(analyzer.masnavi_pos_df[analyzer.masnavi_pos_df['upos']=='NOUN']['text'])
stanza_words_frequencies = nltk.FreqDist(stanza_words).most_common(100)
stanza_words_frequencies[:10]
print ('%-16s' % 'Number of words', '%-16s' % len(stanza_words))
print ('%-16s' % 'Number of unique words', '%-16s' % len(set(stanza_words)))
avg=np.sum([len(word) for word in stanza_words])/len(stanza_words)
print ('%-16s' % 'Average word length', '%-16s' % avg)
print ('%-16s' % 'Longest word', '%-16s' % stanza_words[np.argmax([len(word) for word in stanza_words])])<jupyter_output>Number of words  104110          
Number of unique words 12278           
Average word length 3.7438574584574007
Longest word     اسراییلیانند<jupyter_text>TF/IDF Analysis Using First Tokenization Approach<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import NMF, LatentDirichletAllocation
import matplotlib.pyplot as plt
tfidf_vectorizer = TfidfVectorizer(
    max_df=0.95, min_df=2, max_features=10000
)
tfidf = tfidf_vectorizer.fit_transform(all_words)
tf_vectorizer = CountVectorizer(
    max_df=0.95, min_df=2, max_features=10000
)
tf = tf_vectorizer.fit_transform(all_words)
nmf = NMF(
    n_components=10,
    random_state=1,
    beta_loss="kullback-leibler",
    solver="mu",
    max_iter=1000,
    alpha_W=0.1,
    alpha_H=0.1,
    l1_ratio=0.5,
)
W = nmf.fit_transform(tfidf)
H = nmf.components_
H.shape
tfidf_feature_names = tfidf_vectorizer.get_feature_names_out()
tfidf_feature_names = [make_farsi_text(t) for t in tfidf_feature_names]
plot_top_words(
    nmf,
    tfidf_feature_names,
    10,
    "Topics in NMF model",
)
lda = LatentDirichletAllocation(
    n_components=10,
    max_iter=5,
    learning_method="online",
    learning_offset=50.0,
    random_state=0
)
lda.fit(tf)
tf_feature_names = tf_vectorizer.get_feature_names_out()
tf_feature_names = [make_farsi_text(t) for t in tf_feature_names]
top_words = 10
plot_top_words(lda, tf_feature_names, top_words, "Topics in LDA model")<jupyter_output><empty_output><jupyter_text>TF/IDF Analysis Using Stanza Approach<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import NMF, LatentDirichletAllocation
import matplotlib.pyplot as plt
tfidf_vectorizer = TfidfVectorizer(
    max_df=0.95, min_df=2, max_features=10000
)
tfidf = tfidf_vectorizer.fit_transform(stanza_words)
tf_vectorizer = CountVectorizer(
    max_df=0.95, min_df=2, max_features=10000
)
tf = tf_vectorizer.fit_transform(stanza_words)
nmf = NMF(
    n_components=10,
    random_state=1,
    beta_loss="kullback-leibler",
    solver="mu",
    max_iter=1000,
    alpha_W=0.1,
    alpha_H=0.1,
    l1_ratio=0.5,
)
W = nmf.fit_transform(tfidf)
H = nmf.components_
H.shape
tfidf_feature_names = tfidf_vectorizer.get_feature_names_out()
tfidf_feature_names = [make_farsi_text(t) for t in tfidf_feature_names]
plot_top_words(
    nmf,
    tfidf_feature_names,
    10,
    "Topics with NFM Using Stanza Model",
)
lda = LatentDirichletAllocation(
    n_components=10,
    max_iter=5,
    learning_method="online",
    learning_offset=50.0,
    random_state=0
)
lda.fit(tf)
tf_feature_names = tf_vectorizer.get_feature_names_out()
tf_feature_names = [make_farsi_text(t) for t in tf_feature_names]
top_words = 10
plot_top_words(lda, tf_feature_names, top_words, "Topics with LDA algorithm Using Stanza Model")<jupyter_output><empty_output><jupyter_text>Plot frequent words by Daftar Number<jupyter_code>freq_df = pd.DataFrame(columns=['index', 'Freq', 'Daftar'])
for i in range(1,7):
    freq_dict = nltk.FreqDist(chain(*masnavi_df[masnavi_df['Daftar'] == i]['Couplet_tokenized']))
    freq_dict = pd.DataFrame.from_dict(freq_dict, orient='index', columns=['Freq'])
    freq_dict['Daftar'] = [i] * freq_dict.shape[0]
    freq_dict.reset_index(level=0, inplace=True)
    freq_dict['index'] = freq_dict['index'].apply(lambda x: make_farsi_text(x))
    freq_df = pd.concat([freq_df, freq_dict])
freq_df = freq_df[freq_df['Freq']> 100]
g = sns.FacetGrid(freq_df, col="Daftar", height=4, aspect=1, col_wrap=3)
g.set_xticklabels(rotation=30)
g.map(sns.barplot, "index", "Freq", order=list(set(freq_df['index'])))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13440485464758703, 'fraction_numerical': 0.03312968961417032, 'mean_word_length': 4.9487804878048784, 'pattern_counts': {'":': 1, '<': 224, '<?xml version=': 0, '>': 229, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 12, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '56', 'dir': 'jupyter-structured-clean-dedup', 'id': '571511', 'license': 'MIT', 'n_tokens_mistral': 9952, 'n_tokens_neox': 9328, 'n_words': 1703, 'path': 'preprocess_poem.ipynb', 'repo_name': 'SajjadMb/Masnavi_NLP_Toolkit', 'size': '24606'}