<jupyter_start><jupyter_text>Import required libraries<jupyter_code>#!pip install simpletransformers
from simpletransformers.classification import (
    MultiLabelClassificationModel, MultiLabelClassificationArgs
)
from sklearn.metrics import hamming_loss
import pandas as pd
import numpy as np
from sklearn.metrics import (
    accuracy_score, recall_score, precision_score, f1_score, confusion_matrix, classification_report
)
import warnings
import json
from pandas.core.common import SettingWithCopyWarning
warnings.simplefilter(action="ignore", category=SettingWithCopyWarning)<jupyter_output><empty_output><jupyter_text>Presets- SAVE_MODEL_PATH is the path where the output will be saved after training- DATA_SET_PATH is the path to the github-issues dataset- MAX_TRAINING_EXAMPLES is the amount of issues that we sample randomly, default is 100,000- TRAINING_EPOCHS is the epochs that we run for training, default is 5- labels are the github labels that will be sampled with an exact match- SHOULD_TRAIN is an indicator wether to run the training task or not, if set to False then it means that a pre-trained model is being loaded for evaluation tasks, (requires LOAD_MODEL_PATH to be set)- LOAD_MODEL_PATH is the path of pretraining to use. This can be a path on the disk or name of a default pretraining which is a part of huggingface library which will be automatically downloaded, default is 'roberta-base'<jupyter_code>#PRESETS
SAVE_MODEL_PATH = "outputs" # The directory to save the model to after training
DATA_SET_PATH = 'github-issues.csv'
MAX_TRAINING_EXAMPLES = 100000 # Sampleset
TRAINING_EPOCHS = 5
labels = ["bug", "enhancement", "question"] #The labels that we will be working on

SHOULD_TRAIN = True
LOAD_MODEL_PATH = 'roberta-base'
#Load Data. The dataset contains issues with their labels alongwith a language identifier
#from google.colab import drive
#drive.mount('/content/drive')
issueData = pd.read_csv(DATA_SET_PATH)<jupyter_output><empty_output><jupyter_text>Filtering- Filter out english issues- Make a feature by combining title and body- Add column for each class representing its presence of abence (multi-label mapping)<jupyter_code>#Extracts data that are in 'en' / english
issueData = issueData[issueData['language'] == 'en'].reset_index()


#Concatenate the body and title of the issues to be used as a single feature
issueData['title_body_concat'] = issueData['body'].astype(str) + " " + issueData['title']
issueData['title_body_concat'] = issueData["title_body_concat"].str.lower()

def multiLabelEncode(labelStrings, column):
    outputList = [0]*len(labelStrings)
    for i in range(0,len(labelStrings)):
        labelList = json.loads(labelStrings[i])
        for j in labelList:
            labelname = j['name:']
            labelname=labelname.replace(" ",'').replace(":","").replace("/","").replace("type","").replace("kind","")
            if (labelname == column):
                outputList[i] = 1
                break
    return outputList

#Generate columns for every label (TODO: Iterate over labels to assign these)
issueData = issueData.assign(bug=lambda x: multiLabelEncode(x['labels'], 'bug'))
issueData = issueData.assign(enhancement=lambda x: multiLabelEncode(x['labels'], 'enhancement'))
issueData = issueData.assign(question=lambda x: multiLabelEncode(x['labels'], 'question'))

cols = issueData.columns

#Print the number of instances in our dataset
for label in labels:
    print(label + ": " + str(len(issueData[issueData[label]==1])))

label_count = len(labels)

#Shuffle the dataset randomly
issueData = issueData.sample(frac=1).reset_index(drop=True)<jupyter_output>bug: 117768
enhancement: 87620
question: 17957<jupyter_text>SamplingSample MAX_TRAINING_EXAMPLES randomly form the dataset, then filter out the labels from above<jupyter_code>randomSample = issueData.sample(n = MAX_TRAINING_EXAMPLES)

slashedData = []

for label in labels:
    slashedData.append(randomSample[randomSample[label]==1])

# Make a concatenated dataframe for working
df = pd.concat(slashedData)

for label in labels:
    print(label + ": " + str(len(df[df[label]==1])))<jupyter_output>bug: 18374
enhancement: 13982
question: 3016<jupyter_text>Further preprocessingFurther preprocessing includes resolving any imbalance- First we find the label with the largest sample count- Then for every label we check if it has a sample count that is less than half of the samples of the largest label- If any such label exists, we oversample it equal to the sample set of the largest labelThen we prepare the data by generating khot encoding for the transformer model<jupyter_code># Solve any label imbalance by over sampling #
largest_sampled_label = labels[0]
for label in labels:
    currentLabelSampleCount = len(df[df[label]==1])
    largestLabelSampleCount = len(df[df[largest_sampled_label]==1])
    if (currentLabelSampleCount > largestLabelSampleCount):
        largest_sampled_label = label

#Check if any label has less than half of the largest label's sample count and oversample
largestLabelSampleCount = len(df[df[largest_sampled_label]==1])
for label in labels:
    currentLabelSampleCount = len(df[df[label]==1])
    if (currentLabelSampleCount < (largestLabelSampleCount*0.5)):
        samplesToAdd = largestLabelSampleCount - currentLabelSampleCount
        samplesInOriginalSet = len(issueData[issueData[label]==1])
        if ((samplesToAdd + currentLabelSampleCount) > samplesInOriginalSet):
            samplesToAdd = samplesInOriginalSet - currentLabelSampleCount
        df = df.append(issueData[issueData[label]==1].sample(n = samplesToAdd))
#

#Shuffle the dataframe again
df = df.sample(frac=1).reset_index(drop=True)

total_sample = 0
for label in labels:
    print(label + ": " + str(len(df[df[label]==1])))
    total_sample+= len(df[df[label]==1])

#Convert the labels into one hot encoding vector form
df['k_hot_labels'] = list(df[labels].values)<jupyter_output>bug: 18738
enhancement: 14768
question: 17957<jupyter_text>Split data<jupyter_code>#Split dataframes into two frames for training and testing with a threshold of 0.8, 0.2
df = df[['title_body_concat','k_hot_labels']]
msk = np.random.rand(len(df)) < 0.8
train = df[msk]
test = df[~msk]
train.columns = ["text", "labels"]
test.columns = ["text", "labels"]<jupyter_output><empty_output><jupyter_text>Model training<jupyter_code># Preparing train data
train_df = train
eval_df = test

# Optional model configuration
model_args = MultiLabelClassificationArgs(
    num_train_epochs=TRAINING_EPOCHS,
    overwrite_output_dir=True,
    save_model_every_epoch=False,
    save_eval_checkpoints=False,
    save_steps=-1,
    output_dir=SAVE_MODEL_PATH
)

# Create a MultiLabelClassificationModel


model = MultiLabelClassificationModel(
    "roberta",
    LOAD_MODEL_PATH,
    num_labels=label_count,
    use_cuda=True,
    args = model_args
)


# Train the model
if (SHOULD_TRAIN==True):
    model.train_model(train_df)

# Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(
    eval_df
)

print("LRAP Evaluation:-")
print(result)
# Make predictions with the model
test_list = eval_df['text'].tolist()
test_list_labels = eval_df['labels'].tolist()
predictions, raw_outputs = model.predict(test_list)
result_df = pd.DataFrame(predictions, columns=labels)
original_df = pd.DataFrame(test_list_labels, columns = labels)<jupyter_output><empty_output><jupyter_text>Classification matrix for every label<jupyter_code>for label in labels:
    print("* " + label)
    print("Confusion matrix")
    print(confusion_matrix(original_df[label],result_df[label]))<jupyter_output>* bug
Confusion matrix
[[5564  716]
 [ 699 2988]]
* enhancement
Confusion matrix
[[6223  728]
 [ 723 2293]]
* question
Confusion matrix
[[5739  636]
 [ 715 2877]]<jupyter_text>Print hamming loss for every label<jupyter_code>for label in labels:
    print("Hamming loss " + label + ": " +str(hamming_loss(original_df[label],result_df[label])))<jupyter_output>Hamming loss bug: 0.14196849603692185
Hamming loss enhancement: 0.1455804153707234
Hamming loss question: 0.13554730611016355<jupyter_text>Classification reportPrint all the evaluation metrices for the model<jupyter_code>original_onehot = []
predicted_onehot = []


original_onehot = list(original_df[labels].values)
predicted_onehot = list(result_df[labels].values)
print(classification_report(original_onehot,predicted_onehot, zero_division=0, target_names=labels))<jupyter_output>precision    recall  f1-score   support

         bug       0.81      0.81      0.81      3687
 enhancement       0.76      0.76      0.76      3016
    question       0.82      0.80      0.81      3592

   micro avg       0.80      0.79      0.79     10295
   macro avg       0.79      0.79      0.79     10295
weighted avg       0.80      0.79      0.79     10295
 samples avg       0.80      0.80      0.80     10295
{'fraction_non_alphanumeric': 0.09410571396371013, 'fraction_numerical': 0.03144370562380255, 'mean_word_length': 5.049079754601227, 'pattern_counts': {'":': 5, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '58703', 'license': 'MIT', 'n_tokens_mistral': 2939, 'n_tokens_neox': 2726, 'n_words': 774, 'path': 'model.ipynb', 'repo_name': 'ARiSE-NDSU/maintainomate', 'size': '8873'}