<jupyter_start><jupyter_text>Using regex to filter for comparative threads  filter using VS<jupyter_code>import re

# find all thread title that contains VS (ignore case)
vs = re.compile("vs", re.IGNORECASE)
vs_result = conn.find({"title":vs},{"title":1,"replies.clean_text":1})

# to filter the subfield
# conn.find({ "field.subfield" : "value" })

ids = []
titles = []
replies = []

for idx,val in enumerate(vs_result):
    # add to the lists
    ids.append(val['_id'])
    titles.append(val['title'])
    replies_arr = []
    
    for reply in val['replies'][1:]: # don't want the original post asking the question so use [1:]
        text = reply['clean_text'].replace('\n','') # want to get rid of the \n\n
        replies_arr.append(text)
    replies.append(replies_arr)
titles # looks to be pretty good at catching comparative threads
len(titles)<jupyter_output><empty_output><jupyter_text>- potential problem:    * looks like it might also catch other irrelevant comparison:        * brand comparision        * soft vs hard comparision        * manufactoring country comparision    * duplicate thread title: check for id and replies<jupyter_code>replies[:2] # looking at 2 replies
import pandas as pd
# create pandas dataframe where each reply is a row
vs_df = pd.DataFrame({'id':ids,'thread_title':titles,'reply':replies})
vs_df = replies_df.explode('reply') # explode() to go from item in arr to row
vs_df.head()
def generate_df_from_result(result):
    """
    input: mongodb query result
    output: dataframe
    """
    ids = []
    titles = []
    replies = []

    for idx,val in enumerate(result):
        # add to the lists
        ids.append(val['_id'])
        titles.append(val['title'])
        replies_arr = []

        for reply in val['replies'][1:]: # don't want the original post asking the question so use [1:]
            text = reply['clean_text'].replace('\n','') # want to get rid of the \n\n
            replies_arr.append(text)
        replies.append(replies_arr)
    df = pd.DataFrame({'id':ids,'thread_title':titles,'reply':replies})
    df = df.explode('reply') # explode() to go from item in arr to row
    return df<jupyter_output><empty_output><jupyter_text>filtering using 'OR','/','AND','&'<jupyter_code># find all thread title that contains OR or '/' symbol (ignore case)
regex_or = re.compile("( or )|(\/)", re.IGNORECASE)
regex_or_result = conn.find({"title":regex_or},{"title":1,"replies.clean_text":1})
or_df = generate_df_from_result(regex_or_result)
or_df.head()
or_df.thread_title.unique()<jupyter_output><empty_output><jupyter_text>* Similar problem as using vs<jupyter_code>replies_df = pd.concat([vs_df,or_df])
vs_df.shape
or_df.shape
replies_df.shape
# find all thread title that contains AND or '&' symbol (ignore case)
regex_and = re.compile("( and )|(\&)", re.IGNORECASE)
regex_and_result = conn.find({"title":regex_and},{"title":1,"replies.clean_text":1})
and_df = generate_df_from_result(regex_and_result)
and_df.thread_title.unique()<jupyter_output><empty_output><jupyter_text>* threads less likely to be comparative by using the keyword and, will not combine it - To get a list of potentially comparative threads:    * combine threads with title containing "VS" or "OR" or "/"    * did not use the term "AND" or "&" because high percentage of them not related to direct comparision between 2 rubbers    * replies_df contains the list of potentially comparative threads  Explore comparative threads<jupyter_code>replies_df.thread_title.unique()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12705149971703453, 'fraction_numerical': 0.0036785512167515563, 'mean_word_length': 4.489130434782608, 'pattern_counts': {'":': 9, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '241002', 'license': 'MIT', 'n_tokens_mistral': 1149, 'n_tokens_neox': 1149, 'n_words': 320, 'path': 'EDA/filter_thread/MongoDB_data_exploration.ipynb', 'repo_name': 'vionwinnie/tableTennis-product-review', 'size': '3534'}