<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Load Data1.1&nbsp;&nbsp;Prepare Text Data1.2&nbsp;&nbsp;Show Model Topics1.3&nbsp;&nbsp;Extract Keywords1.3.1&nbsp;&nbsp;Most Frequent Words1.3.2&nbsp;&nbsp;LDA Keywords1.3.3&nbsp;&nbsp;BERT Keywords1.3.4&nbsp;&nbsp;TFIDF Keywords1.4&nbsp;&nbsp;Translate Output1.5&nbsp;&nbsp;Organize by Part of Speech2&nbsp;&nbsp;Visualization Functions2.1&nbsp;&nbsp;Graph of Topic Number Evaluations2.2&nbsp;&nbsp;t-SNE2.3&nbsp;&nbsp;pyLDAvis Topic Visualization2.4&nbsp;&nbsp;Word Cloud3&nbsp;&nbsp;gen_files **kw_extraction**This notebook describes the use of [kwx](https://github.com/andrewtavis/kwx) by deriving the top keywords for tweets from the [Twitter US Airline Sentiment](https://www.kaggle.com/crowdflower/twitter-airline-sentiment) dataset. Follow the provided link and download the data, rename it `airline_tweets.csv` to be more descriptive, and then put it in the directory with this notebook. The .csv is also available in [kwx/examples](https://github.com/andrewtavis/kwx/tree/main/examples).If using this notebook in [Google Colab](https://colab.research.google.com/github/andrewtavis/kwx/blob/main/examples/kw_extraction.ipynb), download and rename the file, then select the `Files` option to the left, and within that use `Upload to session storage`. Colab users can activate GPUs by following `Edit > Notebook settings > Hardware accelerator` and selecting `GPU`. The GPU setup needs to be done before uploading the data.<jupyter_code># pip install kwx -U<jupyter_output><empty_output><jupyter_text>The following gensim update might be necessary in Google Colab as the default version is very low.<jupyter_code># pip install gensim -U
import os
import sys

import numpy as np
import pandas as pd

from kwx.utils import load_data, prepare_data
from kwx.utils import organize_by_pos, translate_output
from kwx.model import extract_kws, gen_files
from kwx.visuals import graph_topic_num_evals, pyLDAvis_topics
from kwx.visuals import gen_word_cloud, t_sne

import matplotlib.pyplot as plt
import seaborn as sns

sns.set(style="darkgrid")
sns.set(rc={"figure.figsize": (15, 5)})

pd.set_option("display.max_rows", 16)
pd.set_option("display.max_columns", None)
from IPython.core.display import display, HTML

display(HTML("<style>.container { width:99% !important; }</style>"))<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>df_airline_tweets = load_data(data="./airline_tweets.csv", target_cols="text")
df_airline_tweets[15:20]<jupyter_output><empty_output><jupyter_text>Prepare Text Data<jupyter_code>input_language, output_language = "english", "english"
text_corpus = prepare_data(
    data=df_airline_tweets,
    target_cols="text",
    input_language=input_language,
    min_token_freq=2,  # 0 for BERT
    min_token_len=4,  # 0 for BERT
    remove_stopwords=True,  # False for BERT
    verbose=True,
)
text_corpus[15:20]<jupyter_output><empty_output><jupyter_text>Show Model Topics<jupyter_code>num_keywords = 15
num_topics = 10
# return_topics=True gives us the topics themselves
topics = extract_kws(
    method="LDA",
    bert_st_model=None,
    text_corpus=text_corpus,
    input_language=input_language,
    output_language=None,
    num_keywords=num_keywords,
    num_topics=num_topics,
    corpuses_to_compare=None,
    return_topics=True,
    ignore_words=None,
    prompt_remove_words=False,
)
topics[0]<jupyter_output><empty_output><jupyter_text>Extract Keywords  Most Frequent Words<jupyter_code># A string or list of strings to not include in outputs
ignore_words = None
freq_kws = extract_kws(
    method="frequency",
    bert_st_model=None,
    text_corpus=text_corpus,
    input_language=input_language,
    output_language=None,
    num_keywords=num_keywords,
    num_topics=num_topics,
    corpuses_to_compare=None,
    ignore_words=None,
    prompt_remove_words=False,
)
freq_kws<jupyter_output><empty_output><jupyter_text>LDA Keywords<jupyter_code># We can pass keywords for gensim.models.ldamulticore.LdaMulticore
lda_kws = extract_kws(
    method="LDA",
    bert_st_model=None,
    text_corpus=text_corpus,
    input_language=input_language,
    output_language=None,
    num_keywords=num_keywords,
    num_topics=num_topics,
    ignore_words=None,
    prompt_remove_words=True,
    passes=20,
    eval_every=5,
    decay=0.5,
)
lda_kws<jupyter_output><empty_output><jupyter_text>BERT Keywords<jupyter_code># Remove n-grams for BERT training
corpus_no_ngrams = [
    " ".join([t for t in text.split(" ") if "_" not in t]) for text in text_corpus
]
# We can pass keywords for sentence_transformers.SentenceTransformer.encode
bert_kws = extract_kws(
    method="BERT",
    bert_st_model="xlm-r-bert-base-nli-stsb-mean-tokens",
    text_corpus=corpus_no_ngrams,
    input_language=input_language,
    output_language=None,
    num_keywords=num_keywords,
    num_topics=num_topics,
    ignore_words=ignore_words,
    prompt_remove_words=True,
    show_progress_bar=True,
    batch_size=32,
)
bert_kws<jupyter_output><empty_output><jupyter_text>TFIDF Keywords The following will look to see if there are words that are specific to United tweets.<jupyter_code>df_united = df_airline_tweets[df_airline_tweets["text"].str.contains("united")]
united_corpus = prepare_data(
    data=df_united,
    target_cols="text",
    input_language=input_language,
    min_token_freq=2,  # 0 for BERT
    min_token_len=4,  # 0 for BERT
    remove_stopwords=True,  # False for BERT
    verbose=True,
)
df_other_airlines = df_airline_tweets.loc[
    np.setdiff1d(df_airline_tweets.index, df_united.index)
]
other_airlines_corpus = prepare_data(
    data=df_other_airlines,
    target_cols="text",
    input_language=input_language,
    min_token_freq=2,  # 0 for BERT
    min_token_len=4,  # 0 for BERT
    remove_stopwords=True,  # False for BERT
    verbose=True,
)
# Words that are prevalent in United tweets compared to others
# We use the corpuses_to_compare argument to compare prevalent words

# We can pass keywords for sklearn.feature_extraction.text.TfidfVectorize
tfidf_kws = extract_kws(
    method="TFIDF",
    text_corpus=united_corpus,
    input_language=input_language,
    output_language=None,
    num_keywords=10,
    num_topics=10,
    corpuses_to_compare=other_airlines_corpus,
    ignore_words=ignore_words,
    prompt_remove_words=False,
    use_idf=True,
    smooth_idf=True,
)
tfidf_kws<jupyter_output><empty_output><jupyter_text>Translate Output<jupyter_code># The Google Translate package doesn't work at times
# translate_output(
#     outputs=lda_kws,
#     input_language=input_language,
#     output_language='spanish'
# )<jupyter_output><empty_output><jupyter_text>Organize by Part of Speech<jupyter_code>organize_by_pos(outputs=lda_kws, output_language=output_language)<jupyter_output><empty_output><jupyter_text>Visualization Functions  Graph of Topic Number Evaluations<jupyter_code>topic_nums_to_compare = list(range(5, 16))
# One extra topic number will be plotted so that all can be compared
figure = graph_topic_num_evals(
    method=["lda", "bert"],
    text_corpus=text_corpus,
    input_language=input_language,
    num_keywords=num_keywords,
    topic_nums_to_compare=topic_nums_to_compare,
    metrics=True,  # stability and coherence
    save_file=False,  # True for pwd or directory name
    verbose=True,
)

plt.show()<jupyter_output><empty_output><jupyter_text>t-SNE<jupyter_code>t_sne(
    dimension="both",
    text_corpus=text_corpus,
    num_topics=num_topics,
    remove_3d_outliers=True,
    fig_size=(20, 10),
    save_file=False,  # True for pwd or directory name
)

plt.show()<jupyter_output><empty_output><jupyter_text>pyLDAvis Topic Visualization<jupyter_code>pyLDAvis_topics(
    method="LDA",
    text_corpus=text_corpus,
    num_topics=num_topics,
    save_file=False,  # True for pwd or directory name
    display_ipython=True,  # <- show in Jupyter notebook
)<jupyter_output><empty_output><jupyter_text>Word Cloud<jupyter_code>ignore_words = [
    "jetblue",
    "united",
    "americanair",
    "usairway",
    "southwestair",
    "virginamerica",
    "fleek",
    "usairways",
    "flightled",
]
gen_word_cloud(
    text_corpus=text_corpus,
    ignore_words=ignore_words,
    height=500,
    save_file=False,  # True for pwd or directory name
)<jupyter_output><empty_output><jupyter_text>gen_files [kwx.model.gen_files](https://github.com/andrewtavis/kwx/blob/main/kwx/model.py) does the following:- Computes the optimal number of topics for the given model type(s)- Extracts the most frequent keywords and those for the optimal topic model- Allows the user to refine keywords given their intuitions- Plots the desired visuals- Puts all of the above in a directory or zipped file<jupyter_code>topic_nums_to_compare = list(range(10, 21))
gen_files(
    method=["LDA", "BERT"],
    text_corpus=text_corpus,
    input_language=input_language,
    output_language=None,
    num_keywords=num_keywords,
    topic_nums_to_compare=topic_nums_to_compare,
    ignore_words=ignore_words,
    prompt_remove_words=True,
    verbose=True,
    org_by_pos=False,  # organize keywords by part of speech
    incl_visuals=[
        "topic_num_evals",
        "word_cloud",
        "pyLDAvis",
    ],  # t_sne not zipping properly
    zip_results=True,
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09508478214209058, 'fraction_numerical': 0.00987336338270015, 'mean_word_length': 5.585865724381626, 'pattern_counts': {'":': 1, '<': 68, '<?xml version=': 0, '>': 69, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '560053', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3401, 'n_tokens_neox': 3295, 'n_words': 657, 'path': 'examples/kw_extraction.ipynb', 'repo_name': 'andrewtavis/kwgen', 'size': '9318'}