<jupyter_start><jupyter_text>StackOverflow Tag PredictorStackOverflow lets us post your queries and the other user can help you with answers. The site uses tags for managing the questions effectively. Here we will be predicting tags for a given question. Tags like C, C++, Python are widely used.<jupyter_code>import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
from ast import literal_eval
import pandas as pd
import numpy as np
from tqdm import tqdm
import re
from collections import defaultdict
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import MultiLabelBinarizer<jupyter_output>[nltk_data] Downloading package stopwords to /content/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>Data loading<jupyter_code># for reading the data
def load_data(dirname):
    # laod the data file
    data = pd.read_csv(dirname, sep='\t')
    # convert string charcter to language syntactic characters if any
    data['tags'] = data['tags'].apply(literal_eval)
    return data
# load training and validation data
train_data = load_data('drive/Colab Notebooks/dataset/train.tsv')
val_data = load_data('drive/Colab Notebooks/dataset/validation.tsv')
train_data[45:50]
# training data
X_train = train_data['title'].values 
y_train = train_data['tags'].values
# validation data
X_val = val_data['title'].values
y_val = val_data['tags'].values
print(X_train.shape)
print(y_train.shape)
print(X_val.shape)
print(y_val.shape)<jupyter_output>(100000,)
(100000,)
(30000,)
(30000,)<jupyter_text>Text PreprocessingWe remove the punctuations, unecessary whitespaces and some other characters<jupyter_code># preprocess text
def preprocess_data(text):
    STOPWORDS = set(stopwords.words('english'))
    # convert to lowercase
    text = text.lower()
    # replace whitespaces and punctuations
    text = re.sub('[/(){}\[\]\|@,;]', ' ', text)
    text = re.sub('[^0-9a-z #+_]', '', text)
    text = ' '.join(word for word in text.split() if word not in STOPWORDS)
    return text
# preprocess the data
X_train = [preprocess_data(text) for text in X_train]
X_val = [preprocess_data(text) for text in X_val]<jupyter_output><empty_output><jupyter_text>Find word and tag frequencies<jupyter_code>def compute_frequency(X_train, y_train):
    # dictionary of all tags with their frequency.
    tag_counts = defaultdict(int)
    # dictionary of all words with their frequency.
    word_counts = defaultdict(int)

    # find tag counts
    for _,tags in tqdm(enumerate(y_train)):
        for tag in tags:
            #print(tag)
            tag_counts[tag] += 1

    # for words
    for _,senten in tqdm(enumerate(X_train)):
        for word in senten.split():
            word_counts[word] += 1
    
    return word_counts, tag_counts
word_counts, tag_counts = compute_frequency(X_train, y_train)<jupyter_output>100000it [00:00, 1006728.80it/s]
100000it [00:00, 419402.72it/s]<jupyter_text>We will create vocabulary dictionary of top **N** words from the training data. We need two mappings:1) Words to index2) Index to words<jupyter_code># for creating word to index and vice versa mappings
def create_vocabulary_mappings(X_train, word_counts, DICT_SIZE=4500):
    # word to index mapping
    word_to_idx = {word:idx for idx,(word,f) in enumerate(
                sorted(word_counts.items(), key=lambda v:v[1], reverse=True)[:DICT_SIZE])}
    # reverse index to word mapping
    idx_to_word= {word_to_idx[word]:word for word in word_to_idx.keys()}
    
    return word_to_idx, idx_to_word
DICT_SIZE=4500
word_to_idx, idx_to_word = create_vocabulary_mappings(X_train, word_counts, DICT_SIZE=4500)<jupyter_output><empty_output><jupyter_text>Now we will be trying two feature representations : Bag of Words(BOW) and TF-IDF. First we will create a function for **BOW**. For BOW we will use most commonly used 4500 words.  Bag of Words<jupyter_code># for creating BOW representation
def create_bag_of_words(text, word_to_idx, DICT_SIZE):
    # Intial Matrix for holding the features
    feature_vector = np.zeros(DICT_SIZE)
    
    # update the word frequencies
    for word in text.split():
        if word in word_to_idx.keys():
            feature_vector[word_to_idx[word]] += 1 
    
    return feature_vector
# create the bag of words feature vector
# we will use a sparse representation , here we will be using csr matrix representation
# for storing it
X_train_bow = sparse.vstack([sparse.csr_matrix(create_bag_of_words(text, word_to_idx, DICT_SIZE)) for text in X_train])
X_val_bow = sparse.vstack([sparse.csr_matrix(create_bag_of_words(text, word_to_idx, DICT_SIZE)) for text in X_val])

print('X_train shape ', X_train_bow.shape)
print('X_val shape ', X_val_bow.shape)<jupyter_output>X_train shape  (100000, 4500)
X_val shape  (30000, 4500)<jupyter_text>TF-IDF<jupyter_code># creates tf-idf feature vector
def create_tfidf_features(X_train, X_val):
    # fit for training data
    tfidf = TfidfVectorizer(ngram_range=(1,2), max_df=0.9, min_df=5, token_pattern='(\S+)')
    # apply for training and validation set
    X_train = tfidf.fit_transform(X_train)
    X_val = tfidf.transform(X_val)
    
    return X_train, X_val, tfidf.vocabulary_
X_train_tfidf, X_val_tfidf, tfidf_vocab = create_tfidf_features(X_train, X_val)
tfidf_reverse_vocab = {i:word for word,i in tfidf_vocab.items()}<jupyter_output><empty_output><jupyter_text>ClassifierSince there can be multiple tags for the query question, so we will represent the output as either 0 or 1, where 1 means that tag is present and 0 means it is absent. So will use MultiLabelBinarizer from scikit-learn for this.<jupyter_code># create an instance
mlb_object = MultiLabelBinarizer(classes=sorted(tag_counts.keys()))
# transform the tags 
y_train = mlb_object.fit_transform(y_train)
y_val = mlb_object.transform(y_val)<jupyter_output><empty_output><jupyter_text>TrainingWe will experiment with different classifiers. We will use One vs All approach here.<jupyter_code>from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression, RidgeClassifier
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn import svm
from sklearn.neighbors import KNeighborsClassifier
import pickle
# define the classifier and fit it to the training data
def train_classifier(X_train, y_train, inner_clf):
    # define the classifier
    clf = OneVsRestClassifier(inner_clf)
    # train it
    clf.fit(X_train, y_train)
    return clf
# classifer for one vs all
ridge_clf = RidgeClassifier()
rf_clf = RandomForestClassifier(n_estimators = 10, max_depth=10, n_jobs=-1, verbose=0)
lr_clf = LogisticRegression(penalty="l2", C=1)
svm_clf = LinearSVC(penalty="l2", C=0.06)
# for bag of words
clf_bow_rf = train_classifier(X_train_bow, y_train, rf_clf)
# for ifidf
clf_tfidf_rf = train_classifier(X_train_tfidf, y_train, rf_clf)
# for bag of words
clf_bow_lr = train_classifier(X_train_bow, y_train, lr_clf)
# for ifidf
clf_tfidf_lr = train_classifier(X_train_tfidf, y_train, lr_clf)

# for bag of words
clf_bow_svm = train_classifier(X_train_bow, y_train, svm_clf)
# for ifidf
clf_tfidf_svm = train_classifier(X_train_tfidf, y_train, svm_clf)

# for bag of words
clf_bow_ridge = train_classifier(X_train_bow, y_train, ridge_clf)
# for ifidf
clf_tfidf_ridge = train_classifier(X_train_tfidf, y_train, ridge_clf)<jupyter_output><empty_output><jupyter_text>Evaluation metrics<jupyter_code>from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.metrics import roc_auc_score 
from sklearn.metrics import average_precision_score
from sklearn.metrics import recall_score
# gives evaluation statistics
def evaluate_classifiers(y_val, predicted):
    print('Accuracy: '+ str(accuracy_score(y_val, predicted)*100))
    print(average_precision_score(y_val, predicted))
# make predictions 
pred_val_bow_ridge = clf_bow_ridge.predict(X_val_bow)
pred_val_tfidf_ridge = clf_tfidf_ridge.predict(X_val_tfidf)
pred_val_bow_rf = clf_bow_rf.predict(X_val_bow)
pred_val_tfidf_rf = clf_tfidf_rf.predict(X_val_tfidf)

pred_val_bow_lr = clf_bow_lr.predict(X_val_bow)
pred_val_tfidf_lr = clf_tfidf_lr.predict(X_val_tfidf)


pred_val_bow_svm = clf_bow_svm.predict(X_val_bow)
pred_val_tfidf_svm = clf_tfidf_svm.predict(X_val_tfidf)

print('SVM')
print('Bag of words')
evaluate_classifiers(y_val, pred_val_bow_svm)
print('Tf-IDF')
evaluate_classifiers(y_val, pred_val_tfidf_svm)
print('Ridge')
print('Bag of words')
evaluate_classifiers(y_val, pred_val_bow_ridge)
print('Tf-IDF')
evaluate_classifiers(y_val, pred_val_tfidf_ridge)
print('Logistic Regression')
print('Bag of words')
evaluate_classifiers(y_val, pred_val_bow_lr)
print('Tf-IDF')
evaluate_classifiers(y_val, pred_val_tfidf_lr)
print('Random Forest')
print('Bag of words')
evaluate_classifiers(y_val, pred_val_bow_rf)
print('Tf-IDF')
evaluate_classifiers(y_val, pred_val_tfidf_rf)<jupyter_output>Random Forest
Bag of words
Accuracy: 0.18333333333333332
0.020434962476275707
Tf-IDF
Accuracy: 0.11666666666666668
0.020050132944348178<jupyter_text>Some Validation results  For TF-IDF<jupyter_code># convert back the predictions to the original tags they are suppose to
pred_val_inverse = mlb_object.inverse_transform(pred_val_tfidf_svm)
# convert the original tag labels
y_val_inverse = mlb_object.inverse_transform(y_val)

for i in range(10):
    print('Query:\t' + str(X_val[i]))
    print('True tags:\t' + str(y_val_inverse[i]))
    print('Predicted tags:\t' + str(pred_val_inverse[i]))
    print()<jupyter_output>Query:	properly return reference class member
True tags:	('eclipse', 'hibernate', 'java', 'web-services')
Predicted tags:	('hibernate', 'java', 'mysql', 'web-services')

Query:	temporarily stop form events either raised handled
True tags:	('ios', 'swift')
Predicted tags:	()

Query:	write program accept number command line display sum digits number using recursive funcion
True tags:	('javascript', 'php')
Predicted tags:	('function', 'javascript', 'php')

Query:	void invalid type variable paint
True tags:	('ruby-on-rails',)
Predicted tags:	('ruby-on-rails',)

Query:	making copy azure database
True tags:	('c#', 'database', 'wpf')
Predicted tags:	('c#', 'wpf')

Query:	redux reducer called
True tags:	('javascript',)
Predicted tags:	('javascript',)

Query:	trying update sql server ce failing
True tags:	('java',)
Predicted tags:	('java',)

Query:	solve orgxmlsaxsaxparseexception linenumber 1 columnnumber 1 content allowed prolog
True tags:	('laravel', 'php')
Predicted tags:	('laravel', 'php')[...]<jupyter_text>For Bag of Words (BOW)<jupyter_code># convert back the predictions to the original tags they are suppose to
pred_val_inverse = mlb_object.inverse_transform(pred_val_bow_svm)
# convert the original tag labels
y_val_inverse = mlb_object.inverse_transform(y_val)

for i in range(10):
    print('Query:\t' + str(X_val[i]))
    print('True tags:\t' + str(y_val_inverse[i]))
    print('Predicted tags:\t' + str(pred_val_inverse[i]))
    print()<jupyter_output>Query:	create swift animation class flips two images
True tags:	('ios', 'swift')
Predicted tags:	('swift',)

Query:	setting default activity android
True tags:	('android', 'java')
Predicted tags:	('android', 'java')

Query:	rspec passing get params put action
True tags:	('ruby-on-rails',)
Predicted tags:	('ruby-on-rails',)

Query:	retrieving sql server output variables c#
True tags:	('asp.net', 'c#')
Predicted tags:	('c#',)

Query:	get value property corresponding sqlalchemy instrumentedattribute
True tags:	('python',)
Predicted tags:	('python',)

Query:	way get guice grapher work
True tags:	('java',)
Predicted tags:	('java',)

Query:	get access violation reading location error code
True tags:	('c++',)
Predicted tags:	('c++',)

Query:	get computed elements table django queryset
True tags:	('django',)
Predicted tags:	('django',)

Query:	add google maps marker firebase data
True tags:	('android', 'google-maps', 'java')
Predicted tags:	('google-maps', 'javascript')

Query:	return permutat[...]
{'fraction_non_alphanumeric': 0.10131285608124845, 'fraction_numerical': 0.017422178185120964, 'mean_word_length': 5.948938611589214, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '77791', 'license': 'MIT', 'n_tokens_mistral': 4466, 'n_tokens_neox': 4227, 'n_words': 1118, 'path': 'StackOverflow Question Tagger.ipynb', 'repo_name': 'susantabiswas/stackoverflow-question-tagger', 'size': '12111'}