<jupyter_start><jupyter_text>CORD-19 Software countingThis notebook is designated to count software mentions based on the CORD19 dataset from: <NAME>.; <NAME> (2021), CORD-19 Software Mentions, Dryad, Dataset, https://doi.org/10.5061/dryad.vmcvdncs0 First, relevant packages must be imported into the notebook.<jupyter_code>import numpy as np
import pandas as pd
import csv
import ast
import collections
import matplotlib.pyplot as plt
import Levenshtein as lev
from fuzzywuzzy import fuzz<jupyter_output><empty_output><jupyter_text>Get the data and save it to a variable.<jupyter_code>CORD19_CSV = pd.read_csv('../data/cord-19/CORD19_software_mentions.csv' , converters={'software': lambda x: x[1:-1].split(',')})<jupyter_output><empty_output><jupyter_text>Show the head of the dataset to inspect all columns and obtain a broad overview.<jupyter_code>CORD19_CSV.head(20)<jupyter_output><empty_output><jupyter_text>The dataset contains nine different columns. Thusly, the next lines of this notebook explore the column "software". Therefore, the column software will be saved to an object.<jupyter_code>software = CORD19_CSV.software
software
len_software_multiple_entries = len(software)
len_software_multiple_entries<jupyter_output><empty_output><jupyter_text>The software object contains 77448 rows. In each row, there are software mentions. Some rows contain more than one mention. For instance, row two and four have two mentions. Consequently, the object needs to be transformed into an object which contains solely one software entry per row.<jupyter_code>software = software.explode(ignore_index = True)<jupyter_output><empty_output><jupyter_text>Remove the brackets around the software mentions.<jupyter_code>software = software.str.replace('\'', '')<jupyter_output><empty_output><jupyter_text>Control the software object and inspect if each row contains only one entry.<jupyter_code>software
len_software_single_entries = len(software)
len_software_single_entries<jupyter_output><empty_output><jupyter_text>Now, the object contains solely one entry per row and has a length of 558792.<jupyter_code>average_entries_per_row = len_software_single_entries/len_software_multiple_entries
average_entries_per_row<jupyter_output><empty_output><jupyter_text>Due to the alignment of the software object, it can be obtained that the dataset contains on average 7.2 software mentions per row. Furthermore, the value_counts function will be used to minimise the number of rows by checking identical duplicates.<jupyter_code>software.value_counts(dropna=False)<jupyter_output><empty_output><jupyter_text>The function value_counts reduced the number of rows to 120279. Nevertheless, the dtype counts same software mentions as distinct. For instance, "SPSS" is listed twice with two varied numbers. For this, the datatype will be converted to a dictionary to check for possible empty spaces.<jupyter_code>software_dict = software.to_dict()
software_dict<jupyter_output><empty_output><jupyter_text>The dictionary contains mentions with space at the first position. This means that the data is not cleaned properly for the value_counts() function. Therefore, the function remove_empty_spaces(d) takes a dictionary and removes spaces at the first position of a string.<jupyter_code>def remove_empty_spaces(dic):
    """ Function removing an empty space at the first position of a string. 
    """
    for i in dic:
        if dic[i][:1] == " ":
            dic[i] = dic[i].strip()
    return dic
software_dict = remove_empty_spaces(software_dict)
software_dict<jupyter_output><empty_output><jupyter_text>Now, the software mentions within the dictionary do not contain empty spaces at the first position of the string. For the use of value_counts(), the dictionary is converted to a pandas series.<jupyter_code>software_series = pd.Series(software_dict)
software_series.value_counts()<jupyter_output><empty_output><jupyter_text>Due to the removal of empty spaces, the length of the object decreased. To further minimise the length of the object, all strings will be capitalized.<jupyter_code>software_dict = software_series.to_dict()
def capitalize_mentions(dic):
    """ Function iterating a dictionary and capitalizing all strings.
    """
    for i in dic:
        dic[i] = dic[i].upper()
    return dic
software_dict = capitalize_mentions(software_dict)
software_dict
software_series = pd.Series(software_dict)
software_series = software_series.value_counts()
software_series<jupyter_output><empty_output><jupyter_text>Due to the capitalization of software mentions, the length of the object could be decreased. Subsequently, the fuzzy-wuzzy compare algorithm will be introduced. This algorithm is based on Levenshtein which checks the similarity of strings by various aspects.<jupyter_code>def fuzzy_ratio_compare(str1, str2, th):
    """ Function to compare to strings based on a given threeshold. 
    """
    ratio = fuzz.ratio(str1, str2)
    if(ratio > th):
        return True
    else:
        return False
    
def fuzzy_partial_ratio_compare(str1, str2, th):
    """ Function to compare to strings based on a given threeshold. 
    """
    ratio = fuzz.partial_ratio(str1, str2)
    if(ratio > th):
        return True
    else:
        return False
    
def fuzzy_token_sort_ratio_compare(str1, str2, th):
    """ Function to compare to strings based on a given threeshold. 
    """
    ratio = fuzz.token_sort_ratio(str1, str2)
    if(ratio > th):
        return True
    else:
        return False

def fuzzy_token_set_ratio_compare(str1, str2, th):
    """ Function to compare to strings based on a given threeshold. 
    """
    ratio = fuzz.token_set_ratio(str1, str2)
    if(ratio > th):
        return True
    else:
        return False<jupyter_output><empty_output><jupyter_text>Due to performance reasons, further investigation will be conducted with a subset of the initial series based on a limit. For this, the successive analysis will focus on the 10% of the data that have the most mentions. Therefore, the corresponding amount of 1/10 of the dataset is calculated.<jupyter_code>ten_percent_position = int(round(len(software_series)*0.1, 0))
ten_percent_position<jupyter_output><empty_output><jupyter_text>The lowest mention within the 10% of most common software is considered for the limit.<jupyter_code>lowest_mention_of_ten_percent = software_series[ten_percent_position]
lowest_mention_of_ten_percent<jupyter_output><empty_output><jupyter_text>As nine is the lowest mention within the 10% of most common software, all mentions with nine or more matches are picked for further analysis.<jupyter_code>software_series_shaped = software_series[software_series >= lowest_mention_of_ten_percent]
selection_limit = len(software_series_shaped)
software_series_shaped<jupyter_output><empty_output><jupyter_text>Converting the series to a DataFrame for comparison purposes. The index of the DataFrame is required for selecting rows.<jupyter_code>ts = software_series_shaped.to_frame()
list_soft = []
list_matches = [0]
for i in range(len(ts)):
    list_soft.append(software_series_shaped.index[i])
    list_matches.append(software_series_shaped[i])
df_shaped = pd.DataFrame()
#Insert the column software and matches
df_shaped['Software'] = list_soft
df_shaped['Matches'] = list_matches[1:]
#Sort the DataFrame by numeric by matches and then alphabetical by software
df_shaped = df_shaped.sort_values(by=['Matches', "Software"])
#Reverse the Dataframe to present the most common software at index position 0 
df_shaped = df_shaped[::-1]  
df_shaped.reset_index(drop = True)<jupyter_output><empty_output><jupyter_text>Replacing special characters to prevent the error "unterminated subpattern" at a later stage of this notebook.<jupyter_code>df_shaped['Software'] = df_shaped.Software.str.replace('(','')
df_shaped['Software'] = df_shaped.Software.str.replace(')','')<jupyter_output><empty_output><jupyter_text>The following function compares software mentions based on the fuzzy-wuzzy method. As a result, a blacklist with identified duplicates and a modified DataFrame is returned.<jupyter_code>def unify_dataframe(df, th):
    """Match software mentions based on fuzzywuzzy algorithm
    """
    df_holder = df
    blacklist = set()
    dict_matches = df.Matches
    for i in range(len(df)):
        for j in range(i + 1, len(df)):
            if df['Software'][i] not in blacklist:
                if(fuzzy_token_set_ratio_compare(df['Software'][i], df['Software'][j], th)):
                    print("Position: "+str(i)+"/"+str(len(df))+" -> '"+df['Software'][i]+"' with " + str(df['Matches'][i]) +
                          " mentions matched with '" + df['Software'][j] + "' mentioned " + str(df['Matches'][j])+ " times")
                    df.iloc[i, df.columns.get_loc('Matches')] = int(df['Matches'].loc[i] + df['Matches'].loc[j])
                    blacklist.add(df['Software'][j])
    df_holder = df
    return df_holder, blacklist
%%time
threeshold = 84
df_returned = unify_dataframe(df_shaped, threeshold)
df_unified = df_returned[0]
blacklist = df_returned[1]<jupyter_output>Position: 0/9351 -> 'R' with 10805 mentions matched with 'R PACKAGE' mentioned 313 times
Position: 0/9351 -> 'R' with 11118 mentions matched with 'R FOUNDATION FOR STATISTICAL COMPUTING' mentioned 280 times
Position: 0/9351 -> 'R' with 11398 mentions matched with 'R CORE TEAM' mentioned 207 times
Position: 0/9351 -> 'R' with 11605 mentions matched with 'R STUDIO' mentioned 148 times
Position: 0/9351 -> 'R' with 11753 mentions matched with 'R DEVELOPMENT CORE TEAM' mentioned 86 times
Position: 0/9351 -> 'R' with 11839 mentions matched with 'R SCRIPT' mentioned 84 times
Position: 0/9351 -> 'R' with 11923 mentions matched with 'MASK R - CNN' mentioned 71 times
Position: 0/9351 -> 'R' with 11994 mentions matched with 'R FOUNDATION FOR STATISTICAL' mentioned 63 times
Position: 0/9351 -> 'R' with 12057 mentions matched with 'R CORE' mentioned 62 times
Position: 0/9351 -> 'R' with 12119 mentions matched with 'R PROJECT FOR STATISTICAL COMPUTING' mentioned 57 times
Position: 0/9351 -> 'R' with[...]<jupyter_text>Next, the DataFrames presents the new aggregation numbers.<jupyter_code>df_unified<jupyter_output><empty_output><jupyter_text>The blacklist contains the matched duplicates which means that they need to be removed from the DataFrame.<jupyter_code>for i in blacklist:
    name_of_index = df_unified[ df_unified['Software'] == i ].index
    df_unified.drop(name_of_index, inplace = True)<jupyter_output><empty_output><jupyter_text>For comparison purposes, the DataFrame is sorted in descending order by matches and then alphabetical by software.<jupyter_code>df_unified = df_unified.sort_values(by=['Matches', "Software"])
#Reverse the Dataframe to present the most common software at index position 0 
df_unified = df_unified[::-1]  
df_unified<jupyter_output><empty_output><jupyter_text>To verify the removal of duplicates, the length of the DataFrame is outputted.<jupyter_code>len_df_unified = len(df_unified)
len_df_unified<jupyter_output><empty_output><jupyter_text>For the highest percentile of software mentiones, the implemented algorithm leads to a reduction of approximately 23.4%.<jupyter_code>reduction = round((1-len_df_unified/selection_limit) * 100, 2)
reduction<jupyter_output><empty_output><jupyter_text>To investigate the position change of software mentions, the following algorithm compares its index position to the sorted position by matches.<jupyter_code>list_change = []
for i in range(len(df_unified)):
    dif = df_unified.index[i]-i
    if(dif > 0):
        list_change.append("+"+str(dif))
    else:
        list_change.append(df_unified.index[i]-i)
df_unified['Change'] = list_change
df_unified.head(20)<jupyter_output><empty_output><jupyter_text>Assigning the outcome of this notebook to a new DataFrame for the classification notebook. The outcome is stored on an external file.<jupyter_code>df_software_mentions = df_unified
df_software_mentions.to_pickle('software_mentions_CS5099.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08723386629111092, 'fraction_numerical': 0.019385303620246874, 'mean_word_length': 5.536004331348132, 'pattern_counts': {'":': 1, '<': 110, '<?xml version=': 0, '>': 128, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '347950', 'license': 'MIT', 'n_tokens_mistral': 3895, 'n_tokens_neox': 3583, 'n_words': 1150, 'path': 'notebooks/CORD-19-software-counting-cs5099.ipynb', 'repo_name': 'kopatri/cs5099admin', 'size': '12086'}