<jupyter_start><jupyter_text>Problem Formulation The problem is a supervised text classification problem, and my goal is to investigate which supervised machine learning methods are best suited to solve it.Given a new issue comes in, I want to assign it to one of the categories (labels). The classifier makes the assumption that each new issue/complaint is assigned to one and only one category. This is a multi-class text classification problem.  Installing Dependencies<jupyter_code>!pip install pandas
!pip3 install numpy
!pip3 install --upgrade scikit-learn
!pip3 install seaborn
!pip3 install sqlalchemy<jupyter_output><empty_output><jupyter_text>Importing Dependencies<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sqlalchemy import create_engine
from io import StringIO
from sklearn import metrics
from sklearn.feature_selection import chi2
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC
from sklearn.metrics import confusion_matrix
from IPython.display import display<jupyter_output><empty_output><jupyter_text>Establishing database connection and reading database information<jupyter_code># To create engine (i.e Engine object), we use create_engine() function of the sqlalchemy package. At its simplest, it accepts the connection string. 
# The connection string provides information about the data source. 

DATABASES = {
    'test_data':{
        'NAME': 'augur',
        'USER': 'augur',
        'PASSWORD': '<PASSWORD>',
        'HOST': 'localhost',
        'PORT': 5434,
    },
}

db = DATABASES['test_data']

engine_string = "postgresql+psycopg2://augur:password@127.0.0.1".format(
    user = db['USER'],
    password = db['PASSWORD'],
    host = db['HOST'],
    port = db['PORT'],
    database = db['NAME'],
)


engine = create_engine(engine_string)<jupyter_output><empty_output><jupyter_text>Data exploration<jupyter_code># read datasets from list of tables in schema
df_issues=pd.read_sql_table('issues',engine,schema='augur_data')
df_issue_labels=pd.read_sql_table('issue_labels',engine,schema='augur_data')
df_issues.head()
df_issue_labels.head()
col_issue_body = df_issues['issue_body']
col_issue_label_text = df_issue_labels['label_text']
col_issue_body.head(20)
col_issue_label_text.head(20)<jupyter_output><empty_output><jupyter_text>For this project, we need only two columns — “label_text” and “issue_body”. Input: issue_body Example: "Fixes bug that causes the PR worker task to fail when inserting PR metadata with null users. Fixes 404" Output: label_text Example: bug<jupyter_code>df = pd.merge(df_issues, df_issue_labels)
df = df.sort_values('label_color', ascending=True)
df = df.drop_duplicates(subset='issue_body', keep='first')
df.shape<jupyter_output><empty_output><jupyter_text>Before diving into training machine learning models, let’s take a look at some examples first and the number of complaints in each class:<jupyter_code>df.head()
df.info<jupyter_output><empty_output><jupyter_text>We will remove missing values in the “issue_body” column, and add a column encoding the product as an integer because categorical variables are often better represented by integers than strings.Also, I will create a couple of dictionaries for future use.<jupyter_code>col = ['label_text', 'issue_body']
df = df[col]
df = df[pd.notnull(df['issue_body'])]
df.columns = ['label_text', 'issue_body']

df.columns = ['label_text', 'issue_body']
df['category_id'] = df['label_text'].factorize()[0]
category_id_df = df[['label_text', 'category_id']].drop_duplicates().sort_values('category_id')
category_to_id = dict(category_id_df.values)
id_to_category = dict(category_id_df[['category_id', 'label_text']].values)<jupyter_output><empty_output><jupyter_text>After cleaning up, this is the first five rows of the data we will be working on:<jupyter_code>df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Let’s check the average max **category_id** and once we plot it we can observe that the Average Category Id is fluctuating (between 5 and 10, and 20)<jupyter_code>plt.figure(figsize=(15,10))
plt.tight_layout()
sns.distplot(df['category_id'])<jupyter_output><empty_output><jupyter_text>Imbalanced Classes We see that the number of issues per label is imbalanced. Users’ complaints are more biased towards enhancement, bug and help wanted.<jupyter_code>fig = plt.figure(figsize=(8,6))
df.groupby('label_text').issue_body.count().plot.bar(ylim=0)
plt.show()<jupyter_output><empty_output><jupyter_text>When we encounter such problems, we are bound to have difficulties solving them with standard algorithms. Conventional algorithms are often biased towards the majority class, not taking the data distribution into consideration. In the worst case, minority classes are treated as outliers and ignored. For some cases, such as fraud detection or cancer prediction, we would need to carefully configure our model or artificially balance the dataset, for example by undersampling or oversampling each class.However, in our case of learning imbalanced data, the majority classes might be of our great interest. It is desirable to have a classifier that gives high prediction accuracy over the majority class, while maintaining reasonable accuracy for the minority classes. Therefore, I will leave it as it is for the time being.  Text Representation<jupyter_code>tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5, norm='l2', encoding='latin-1', ngram_range=(1, 2), stop_words='english')

features = tfidf.fit_transform(df.issue_body).toarray()
labels = df.category_id
features.shape<jupyter_output><empty_output><jupyter_text>I will use sklearn.feature_selection.chi2 to find the terms that are the most correlated with each of the label texts:<jupyter_code>N = 2
for label_text, category_id in sorted(category_to_id.items()):
    features_chi2 = chi2(features, labels == category_id)
    indices = np.argsort(features_chi2[0])
    feature_names = np.array(tfidf.get_feature_names())[indices]
    unigrams = [v for v in feature_names if len(v.split(' ')) == 1]
    bigrams = [v for v in feature_names if len(v.split(' ')) == 2]
    print("# '{}':".format(label_text))
    print("  . Most correlated unigrams:\n. {}".format('\n. '.join(unigrams[-N:])))
    print("  . Most correlated bigrams:\n. {}".format('\n. '.join(bigrams[-N:])))<jupyter_output># 'API Request':
  . Most correlated unigrams:
. 000z
. 00
  . Most correlated bigrams:
. use case
. pull requests
# 'CHAOSS':
  . Most correlated unigrams:
. cii
. sign
  . Most correlated bigrams:
. chaoss augur
. pull request
# 'Learning Python':
  . Most correlated unigrams:
. npm
. vue
  . Most correlated bigrams:
. github com
. pull request
# 'Metrics Candidate Release':
  . Most correlated unigrams:
. candidate
. 2019
  . Most correlated bigrams:
. https chaoss
. 24th 2019
# 'NDE':
  . Most correlated unigrams:
. script
. vagrant
  . Most correlated bigrams:
. gmail com
. config json
# 'backend':
  . Most correlated unigrams:
. vagrant
. githubapi
  . Most correlated bigrams:
. ubuntu 16
. make install
# 'bug':
  . Most correlated unigrams:
. graphs
. bash
  . Most correlated bigrams:
. lib python3
. local lib
# 'critical':
  . Most correlated unigrams:
. called
. fixed
  . Most correlated bigrams:
. error expected
. gmail com
# 'dependencies':
  . Most correlated unigrams:
. re[...]<jupyter_text>Multi-Class Classifier: Features and Design To train supervised classifiers, I first transformed the “issue_body” into a vector of numbers. Then, explored vector representations such as TF-IDF weighted vectors.After having this vector representations of the text we can train supervised classifiers to train unseen “issue_body” and predict the “label_text” on which they fall.After all the above data transformation, now that I have all the features and labels, it is time to train the classifiers. There are a number of algorithms we can use for this type of problem. Naive Bayes Classifier: the one most suitable for word counts is the multinomial variant:<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(df['issue_body'], df['label_text'], random_state = 0)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf = MultinomialNB().fit(X_train_tfidf, y_train)<jupyter_output><empty_output><jupyter_text>After fitting the training set, let’s make some predictions:<jupyter_code>print(clf.predict(count_vect.transform(["This PR implements a few changes to the Docker database setup and adds the initial configuration for a testing database to be distributed with Docker. It also simplifies the logic of reading the `augur.config.json` file as well simplifying the process of exporting a user's configuration."])))
df[df['issue_body'] == "This PR implements a few changes to the Docker database setup and adds the initial configuration for a testing database to be distributed with Docker. It also simplifies the logic of reading the `augur.config.json` file as well simplifying the process of exporting a user's configuration."]
print(clf.predict(count_vect.transform(["Wrote tests for methods used across multiple workers Segregated the `dump_queue` method from the workers. Also enforced the DRY principle(the same method was written 3 times across 3 worker classes)."])))
df[df['issue_body'] == "Wrote tests for methods used across multiple workers Segregated the `dump_queue` method from the workers. Also enforced the DRY principle(the same method was written 3 times across 3 worker classes)."]<jupyter_output><empty_output><jupyter_text>Note: However, the prediction results are not very correct. I will try finding another algorithm for solving text classification problem.  Model Selection Now ready to experiment with different machine learning models, evaluate their accuracy and find the source of any potential issues.I will benchmark the following four models:- Logistic Regression- (Multinomial) Naive Bayes- Linear Support Vector Machine- Random Forest<jupyter_code>models = [
    RandomForestClassifier(n_estimators=200, max_depth=3, random_state=0),
    LinearSVC(),
    MultinomialNB(),
    LogisticRegression(random_state=0),
]
CV = 5
cv_df = pd.DataFrame(index=range(CV * len(models)))
entries = []
for model in models:
    model_name = model.__class__.__name__
    accuracies = cross_val_score(model, features, labels, scoring='accuracy', cv=CV)
    for fold_idx, accuracy in enumerate(accuracies):
        entries.append((model_name, fold_idx, accuracy))

cv_df = pd.DataFrame(entries, columns=['model_name', 'fold_idx', 'accuracy'])

sns.boxplot(x='model_name', y='accuracy', data=cv_df)
sns.stripplot(x='model_name', y='accuracy', data=cv_df, 
              size=8, jitter=True, edgecolor="gray", linewidth=2)
plt.show()
cv_df.groupby('model_name').accuracy.mean()<jupyter_output><empty_output><jupyter_text>Model Evaluation Continue with our best model (LinearSVC),I'm going to look at the confusion matrix, and show the discrepancies between predicted and actual labels.<jupyter_code>model = LinearSVC()
X_train, X_test, y_train, y_test, indices_train, indices_test = train_test_split(features, labels, df.index, test_size=0.33, random_state=0)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

conf_mat = confusion_matrix(y_test, y_pred)
fig, ax = plt.subplots(figsize=(10,10))
sns.heatmap(conf_mat, annot=True, fmt='d',
            xticklabels=category_id_df.label_text.values, yticklabels=category_id_df.label_text.values)

plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()<jupyter_output><empty_output><jupyter_text>From the result we can see that Learning Python has biggest priority among other categories :D<jupyter_code>for predicted in category_id_df.category_id:
    for actual in category_id_df.category_id:
        if predicted != actual :
            print("'{}' predicted as '{}' : {} examples.".format(id_to_category[actual], id_to_category[predicted], conf_mat[actual, predicted]))
            print('')<jupyter_output>'help wanted' predicted as 'Learning Python' : 0 examples.

'dependencies' predicted as 'Learning Python' : 0 examples.

'workers' predicted as 'Learning Python' : 0 examples.

'python' predicted as 'Learning Python' : 1 examples.

'documentation' predicted as 'Learning Python' : 0 examples.

'backend' predicted as 'Learning Python' : 0 examples.

'NDE' predicted as 'Learning Python' : 1 examples.

'good first issue' predicted as 'Learning Python' : 0 examples.

'metric' predicted as 'Learning Python' : 0 examples.

'enhancement' predicted as 'Learning Python' : 0 examples.

'usability' predicted as 'Learning Python' : 0 examples.

'focus area' predicted as 'Learning Python' : 0 examples.

'CHAOSS' predicted as 'Learning Python' : 0 examples.

'Metrics Candidate Release' predicted as 'Learning Python' : 0 examples.

'wontfix' predicted as 'Learning Python' : 0 examples.

'API Request' predicted as 'Learning Python' : 0 examples.

'question' predicted as 'Learning Python' : 0 examples.
[...]<jupyter_text>Again, using the chi-squared test to find the terms that are the most correlated with each of the categories<jupyter_code>model.fit(features, labels)
N = 2
for label_text, category_id in sorted(category_to_id.items()):
    indices = np.argsort(model.coef_[category_id])
    feature_names = np.array(tfidf.get_feature_names())[indices]
    unigrams = [v for v in reversed(feature_names) if len(v.split(' ')) == 1][:N]
    bigrams = [v for v in reversed(feature_names) if len(v.split(' ')) == 2][:N]
    print("# '{}':".format(label_text))
    print("  . Top unigrams:\n       . {}".format('\n       . '.join(unigrams)))
    print("  . Top bigrams:\n       . {}".format('\n       . '.join(bigrams)))
print(df['label_text'].unique())
print(len(df['label_text'].unique()))

print('length of y_test: ', len(y_test))
print('length of y_pred: ', len(y_pred))

print('y_test: ', y_test)
print('y_pred: ', y_pred)

print('labels length ', len(labels.unique()))

# print('target_names length ', len(target_names))

# print(metrics.classification_report(y_test, y_pred, target_names=df['label_text'].unique()))<jupyter_output>['Learning Python' 'help wanted' 'dependencies' 'workers' 'python'
 'documentation' 'backend' 'NDE' 'good first issue' 'metric' 'enhancement'
 'usability' 'focus area' 'CHAOSS' 'Metrics Candidate Release' 'wontfix'
 'API Request' 'question' 'bug' 'critical' 'invalid' 'frontend'
 'deployed version' 'use case' 'installation']
25
length of y_test:  99
length of y_pred:  99
y_test:  271    10
439    10
470     1
2      19
121    10
       ..
95      2
532    14
263    21
327    10
338    10
Name: category_id, Length: 99, dtype: int64
y_pred:  [10 10 14 10 10  1 18 12 14 10  6  2  6 10 12 18 18  7 10 10 10 15 21 10
 10  7  5 21 18  6 10  1 18 10 18 10 10 21 10 18 18 14 18 14 10 18 10 21
 21 10  3  6 10 10 10 10 18 21 18 23 10 18 10 10 21  2  5 10 10 10 18  7
 18  1 18 21 10 10 10 16 21 18  7 10  7 18 10  5 10 14  9 10  6 10  2 14
 21  5 10]
labels length  25<jupyter_text>Graphical representation of comparision y_test to y_pred<jupyter_code>sns.barplot(x=y_pred,y=y_test.values)
plt.title('Bar Plot of y_pred and y_test values',fontsize=20)
plt.show()
df_copy = pd.DataFrame({'Actual': y_test.values.flatten(), 'Predicted': y_pred.flatten()})
df_copy<jupyter_output><empty_output><jupyter_text>Finally, we print out the report for actual and predicted test reuslts<jupyter_code>df_copy = df_copy.head(25)
df_copy.plot(kind='line',figsize=(16,10))
plt.grid(which='major', linestyle='-', linewidth='0.2', color='green')
plt.grid(which='minor', linestyle=':', linewidth='1.5', color='black')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10300111152278622, 'fraction_numerical': 0.021859948128936643, 'mean_word_length': 5.403716884143931, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '540680', 'license': 'MIT', 'n_tokens_mistral': 5435, 'n_tokens_neox': 4967, 'n_words': 1656, 'path': 'notebooks/Nodira-Ibrogimova-text-classification.ipynb', 'repo_name': 'pratikmishra356/augur', 'size': '16189'}