<jupyter_start><jupyter_text>CORD-19 Software classification This notebook is designated to classify software mentions based on the CORD19 dataset from:<NAME>.; <NAME> (2021), CORD-19 Software Mentions, Dryad, Dataset, https://doi.org/10.5061/dryad.vmcvdncs0 First, relevant packages must be imported into the notebook.<jupyter_code>import numpy as np
import pandas as pd
import csv
import ast
import collections
import matplotlib.pyplot as plt
import Levenshtein as lev
from fuzzywuzzy import fuzz 
from matplotlib.ticker import FormatStrFormatter, StrMethodFormatter
import json<jupyter_output><empty_output><jupyter_text>The outcome "df_software_mentions" of the notebook "CORD-19-software-counting-cs5099.ipynb" will be used for classification purposes. Therefore, the notebook reads the content from the file "software_mentions.pkl".<jupyter_code>df_software_mentions = pd.read_pickle('software_mentions_CS5099.pkl')
df_software_mentions<jupyter_output><empty_output><jupyter_text>Shift the focus to the column software and create a column for classification.<jupyter_code>df_software = df_software_mentions.drop('Change', 1)
df_software = df_software.reset_index()
df_software = df_software.drop('index', 1)
df_software['Classification'] = "Unclassified"
df_software<jupyter_output><empty_output><jupyter_text>Subsequently, the next cell outputs software mentions which must be manually copied to the file "software_categories_CS5099.csv" for classification purposes.<jupyter_code>result = df_software.to_json(orient='records')
parsed = json.loads(result)
software_json = json.dumps(parsed, indent=4) 
df_read_json = pd.read_json(software_json)
print(df_read_json.to_string())<jupyter_output>Software  Matches Classification
0                                               R    13163   Unclassified
1                                            SPSS    11290   Unclassified
2                                  GRAPHPAD PRISM     8499   Unclassified
3                                           BLAST     6711   Unclassified
4                                           EXCEL     4319   Unclassified
5                                           STATA     4048   Unclassified
6                                            MEGA     3428   Unclassified
7                                             SAS     3399   Unclassified
8                                          IMAGEJ     2779   Unclassified
9                                          MATLAB     2710   Unclassified
10                                 GOOGLE SCHOLAR     2485   Unclassified
11                                            NET     2411   Unclassified
12                                    [...]<jupyter_text>Next, the file "software_categories_CS5099.csv" is read from the directory and presented.<jupyter_code>Categories_CSV = pd.read_csv('software_categories_CS5099.csv')
Categories_CSV
def get_category(mention):
    """
    Function receiving a software mention a returning its category. When no category is found 'None' is returned.
    The function works dynamically to the entries of the Categoies_CSV.
    """
    category_holder = "None"
    len_categories = len(Categories_CSV.columns)
    i = 0
    while i < len_categories: 
        column_holder = Categories_CSV.columns[i]
        if(any(Categories_CSV[column_holder] == mention) == True):
            return Categories_CSV.columns[i]
        i = i + 1<jupyter_output><empty_output><jupyter_text>Each software mention must be assigned to a category.<jupyter_code>%%time
dict_categories = {}
for i, row in df_software.iterrows():
    row.Classification = get_category(row.Matches)
    dict_categories[i] = get_category(row.Software)
df_software.Classification = dict_categories.values()
df_software.head(5)<jupyter_output>Wall time: 36.1 s<jupyter_text>Consequently, the software categories are summed up and presented.<jupyter_code>len_df_classification_holder = len(df_software)
classification_series = df_software['Classification'].value_counts()
len_classification_series = len(classification_series.index)

df_total_matches = pd.DataFrame(columns=['Matches'], index = classification_series.index )
df_total_matches['Matches'] = 0

i = 0
while i < len_classification_series:
    x = 0
    while x < len_df_classification_holder:
        if df_software['Classification'][x] == classification_series.index[i]:
            df_total_matches['Matches'][classification_series.index[i]] = df_total_matches['Matches'][classification_series.index[i]] + df_software['Matches'][x]
        x = x + 1
    i = i + 1

df_total_matches.sort_values(by="Matches", ascending=False)
df_total_matches_sorted = df_total_matches.sort_values(by="Matches", ascending=True)

x = df_total_matches_sorted['Matches'].index
y = df_total_matches_sorted['Matches'].values

fig, ax = plt.subplots()
plt.barh(x, y)

# x axis with commata 
ax.xaxis.set_major_formatter(StrMethodFormatter('{x:,}'))

# labelling the bars
for i, v in enumerate(y):
    ax.text(v + 1000, i - 0.1, str(f'{v:,}'), color='black', backgroundcolor='white')

plt.title("Software mentions per category")
plt.xlabel("Matches")
plt.ylabel("Category")
%config InlineBackend.figure_format = 'svg'
plt.savefig('software_mentions_plot.svg',bbox_inches='tight', dpi=400)  

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08541666666666667, 'fraction_numerical': 0.024242424242424242, 'mean_word_length': 3.2865259740259742, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '168866', 'license': 'MIT', 'n_tokens_mistral': 1666, 'n_tokens_neox': 1513, 'n_words': 364, 'path': 'notebooks/CORD-19-software-classification-cs5099.ipynb', 'repo_name': 'kopatri/cs5099admin', 'size': '5295'}