<jupyter_start><jupyter_text>Introduction This module helps you extract key terms and topics from corpus using a comparative approach.  Installation  Usage  Import packages<jupyter_code>from compExtract import ComparativeExtraction<jupyter_output><empty_output><jupyter_text>Load sample data<jupyter_code>import pandas as pd
import numpy as np
PATH = "/Users/xiaoma/Desktop/gitrepo/associate-term-search/data/switch_reviews.csv"
data = pd.read_csv(PATH)
label = [x <= 3 for x in data['stars']]
data
data.columns<jupyter_output><empty_output><jupyter_text>Here we are using online Amazon reviews for Nintendo Switch to illustrate the usages of the module. The module requires a corpus and a set of binary labels as inputs. The labels should be created depending on what type of questions are we trying to answer. The set of labels should be of the same length as the corpus.Here, let's assume that we want to know why people dislike this product and find relevant keywords. To answer this question, we created the label to be a binary variable indicating whether a reviewer gives a 3 star or less.   Initialize the module with the review corpus and labels<jupyter_code>ce = ComparativeExtraction(corpus = data['reviews'], labels = label)<jupyter_output><empty_output><jupyter_text>Extract the keywords<jupyter_code>ce.get_distinguish_terms(ngram_range = (1,3),top_n = 10)
# Get the keywords that are mentioned significantly more in the less than or equal to 3 star reviews
ce.increased_terms_df
# Get the keywords that are mentioned significantly less in the less than or equal to 3 star reviews
ce.decreased_terms_df<jupyter_output><empty_output><jupyter_text>If we need more context on a given word, or we need more interpretable topics, we can:1. Output the reviews that contains the term2. Switch the ngram_range  Output the reviews Say we want to know more about the significant term "work", we can directly output all the reviews containing the term.The output class "kw" contains a one-hot encoded document-term-matrix that has all the terms found from the corpus. We can leverage it to find corresponding reviews of each term.<jupyter_code># The binary_dtm provides a convenient way to extract reviews with specific terms
print(ce.binary_dtm[['work']])
reviews_contain_term_work = data['reviews'][[x == 1 for x in ce.binary_dtm['work']]]
len(reviews_contain_term_work)
for x in pd.Series(reviews_contain_term_work).sample(1):
    print(x)<jupyter_output>I bought this as a Christmas present for my son.  After about a month and half of using it.  The switch stopped working.  It wont charge.  The product is an expensive piece of junk.<jupyter_text>Change the n-gram range to exclude uni-grams<jupyter_code>ce_ngram = ComparativeExtraction(corpus = data['reviews'], labels = label).get_distinguish_terms(ngram_range=(2,4), top_n=10)
ce_ngram.increased_terms_df
ce_ngram.decreased_terms_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06461538461538462, 'fraction_numerical': 0.00547008547008547, 'mean_word_length': 5.950118764845605, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '441023', 'license': 'MIT', 'n_tokens_mistral': 869, 'n_tokens_neox': 828, 'n_words': 347, 'path': 'notebook/readme.ipynb', 'repo_name': 'ALaughingHorse/comparative_keyword_extraction', 'size': '2925'}