<jupyter_start><jupyter_text>Filter out the remaining unwanted postsDue to the fact that I trained the classification model to an accuracy of 86%, I will be left with roughly 14% unwanted posts in my dataset. These unwanted posts related to languages such as c, scala and frameworks/ide's such as android and eclipseThis notebook will document the following approach:1. Gather two lists:    * All programming languages    * All frameworks and IDE's related to JAVA2. Iterate through each of the posts that have been filtered using the classifier     * if any words in either of the lists appear in the list of tags associated with a post drop the given row3. Save the resulting dataframe as a CSV  Filtering Process  1) Loading programming languages and frameworks lists<jupyter_code>java_frameworks_list = []
with open('./data/java_frameworks.txt') as file:
    for line in file:
        line = line.replace(' ', '')
        java_frameworks_list.append(line.lower().strip())
program_lang_list = []
with open('./data/programming_languages_list.txt') as file:
    for line in file:
        line = line.replace('-', '').replace(' ', '').replace('/', '').replace('(', '').replace(')', '').strip()
        program_lang_list.append(line.lower())<jupyter_output><empty_output><jupyter_text>2) Remove all posts that include the tags in both lists above (alternative languages to java, related to ide's frameworks)<jupyter_code>import pandas as pd
df = pd.read_csv('./data/filtered_data_ready_for_app.csv')
df.shape
count = 0
for index, row in df.iterrows():
    found = False
    #extract tags from each row
    post_tags = df.loc[index, 'Tags'].split()
    #check if the tags are present in stop lists (frameworks, other languages)
    for tag in post_tags:
        tag = tag.replace('-', '')
        #if an unwanted tag found - remove from the dataframe
        if tag in program_lang_list or tag in java_frameworks_list:
            found = True
    if found:
        df.drop(index, inplace=True)
        #df.drop(df.index[i], inplace=True)
df.to_csv('./data/filtered_cleaned_posts_no_frameworks_no_alt_lang.csv', index=False)
#size of resulting dataset
df.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09780621572212066, 'fraction_numerical': 0.004570383912248629, 'mean_word_length': 4.032183908045977, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '431833', 'license': 'MIT', 'n_tokens_mistral': 641, 'n_tokens_neox': 612, 'n_words': 239, 'path': 'Phase_3B_Filter_out_remaining_unwanted_posts.ipynb', 'repo_name': 'c14410312/StackoverflowPostClassification', 'size': '2188'}