<jupyter_start><jupyter_text>Inferring citation style from the reference string<NAME>16.11.2018This notebook contains the experiments related to inferring citation style from the reference string. The main goal is to build a supervised classifier able to classify the given reference string into one of the known styles.<jupyter_code>import sys
sys.path.append('..')

%matplotlib inline

import warnings
warnings.simplefilter('ignore')

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import re
import seaborn as sns

from config import STYLES
from dataset import add_noise, clean_data, read_ref_strings_data, remove_technical_parts, \
    rearrange_tokens, generate_unknown
from evaluation import evaluate_cv
from features import get_features, get_tfidf_features, select_features_chi2, select_features_rf, \
    tokens_to_classes
from random import randint, seed
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import chi2
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from statistics import mean<jupyter_output>/home/dtkaczyk/.local/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d<jupyter_text>Preparing the data Reading the dataWe are considering the following 17 styles in these experiments:<jupyter_code>STYLES<jupyter_output><empty_output><jupyter_text>The data for the experiments was generated automatically by formatting 5k metadata records (randomly chosen from the Crossref collection) into 17 styles. This resulted in 85k pairs (reference string, citation style name). We also keep the DOI of the metadata record used to generate the reference string.<jupyter_code>dataset = read_ref_strings_data('../data/dataset/')
print('Dataset size: {}'.format(dataset.shape[0]))
dataset.head()<jupyter_output>Dataset size: 85000<jupyter_text>Let's see example entries for each style:<jupyter_code>seed(10)
examples = dataset.groupby('style').nth(randint(0, 4999)).reset_index()
print('\n\n'.join(['{}\n{}'.format(s, t) for s, t in zip(examples['style'].tolist(),
                                                         examples['string'].tolist())]))<jupyter_output>acm-sig-proceedings
[1]<NAME>. et al. 1990. Effect of carbonyl iron powder additions on the properties of materials produced by dynamic hot pressing. Soviet Powder Metallurgy and Metal Ceramics. 29, 6 (Jun. 1990), 427–430.

american-chemical-society
(1) <NAME>.; <NAME>.; <NAME>. Soviet Powder Metallurgy and Metal Ceramics 1990, 29, 427–430.

american-chemical-society-with-titles
(1) <NAME>.; <NAME>.; <NAME>. Effect of Carbonyl Iron Powder Additions on the Properties of Materials Produced by Dynamic Hot Pressing. Soviet Powder Metallurgy and Metal Ceramics 1990, 29, 427–430.

american-institute-of-physics
1 <NAME>, <NAME>, and <NAME>, Soviet Powder Metallurgy and Metal Ceramics 29, 427 (1990).

american-sociological-association
<NAME>., <NAME>, and <NAME>. 1990. “Effect of Carbonyl Iron Powder Additions on the Properties of Materials Produced by Dynamic Hot Pressing.” Soviet Powd[...]<jupyter_text>Data cleaningFirst, let's get rid of null and very short reference strings:<jupyter_code>seed(0)
dataset = clean_data(dataset)
print('Dataset size: {}'.format(dataset.shape[0]))<jupyter_output>Dataset size: 82834<jupyter_text>We also have some remains from DOI stripping and other "technical" fragments in the strings. We will remove those parts.<jupyter_code>dataset['string'] = dataset['string'].apply(remove_technical_parts)<jupyter_output><empty_output><jupyter_text>Reference strings generated automatically are also a bit too perfect. For example, all strings of american-chemical-society style are prefixed by "(1)". This prefix alone would be enough for the learning algorithm to assign a string to this style. On the other hand, we cannot expect all strings "in real life" to have a proper prefix (some tools/extractors might strip this).Also, the strings were generated from rich metadata records, and as a result, contain fragments such as the names of the months from the publication date. In practise, however, the dates are typically given as years only ("(1997)" instead of "(Mar 1997)").To make the dataset a bit less perfect, let's remove such fragments from some of the references:<jupyter_code>dataset['string'] = dataset['string'].apply(add_noise)
dataset.head()<jupyter_output><empty_output><jupyter_text>Finally, we will add the "unknown" category to the dataset. The reference strings in this category will be generated by randomly swapping tokens in the "proper" reference strings.<jupyter_code>dataset_unknown = generate_unknown(dataset, 5000, random_state=0)
dataset = pd.concat([dataset, dataset_unknown])
print('Dataset size: {}'.format(dataset.shape[0]))<jupyter_output>Dataset size: 87834<jupyter_text>Splitting the dataWe will split the dataset randomly into training (75%) and test (25%) sets. The training set will be used to find the best learning algorithm and its meta-parameters. The test set will be used once to evaluate the final chosen model. This provides a fair, unbiased estimate of the model performance.The data will be split based on DOIs, for better separation (i.e. all strings generated from a specific DOI will belong either to the train set or the test set).<jupyter_code>train_doi, test_doi = train_test_split(dataset['doi'].drop_duplicates(), random_state=0)
train_dataset = dataset.loc[dataset['doi'].isin(train_doi)]
test_dataset = dataset.loc[dataset['doi'].isin(test_doi)]
print('Train dataset size: {}'.format(train_dataset.shape[0]))
print('Test dataset size: {}'.format(test_dataset.shape[0]))<jupyter_output>Train dataset size: 65723
Test dataset size: 22111<jupyter_text>Feature representationWe will use a TF-IDF bag-of-ngrams representation of the reference strings. In this case, however, the tokens will be different from the typical use case. We will leave punctuation characters as tokens, and map the actual words into word classes such as "capitalized word", or "uppercase letter". This is done to preserve the style-related information. We will also mark the beginning and the end of the string with special tokens.This function maps the tokens in the string into our token classes:<jupyter_code>tokens_to_classes('<NAME>. Yearbook of Surgery 2006, 2006, 322–324.')<jupyter_output><empty_output><jupyter_text>The basic modelLet's now build some models and evaluate them. We will use logistic regression algorithm to predict the style from TF-IDFs of the tokens. We will use all {1,2}-grams we have in the train set. We will perform a 5-fold cross-validation on the train set, and look at the resulting accuracy.Function *get_tfidf_features* passed to the evaluation function calculates a matrix of TD-IDF features for the list of strings. It is typically called twice, first on the training set and then on the test set (in this case, it will be called twice for every cross-validation split). When called on the training set, it calculates the vocabulary and IDFs using the entire training set (this is called fitting). When called on the test set, nothing is learned from the entire test set, but we reuse the vocabulary and IDFs calculated previously on the training set.<jupyter_code>accuracies, _ = evaluate_cv(train_dataset, LogisticRegression(random_state=0),
                            feature_fun=get_tfidf_features, ngrams=(1, 2), nfeatures=None,
                            feature_selector=None)
print('Fold accuracies: {}'.format(accuracies))
print('Mean accuracy: {}'.format(mean(accuracies)))<jupyter_output>Fold accuracies: [0.9077513120710536, 0.9201824401368301, 0.9118426416632694, 0.9150175791059769, 0.9160464423935695]
Mean accuracy: 0.9141680830741399<jupyter_text>The accuracy is already decent. Choosing the ngram rangeLet's now see how the ngram range affects the accuracy:<jupyter_code>results = np.zeros((4, 4))
for min_r in range(1, 5):
    for max_r in range(min_r, 5):
        accuracies, _ = evaluate_cv(train_dataset, LogisticRegression(random_state=0),
                                    feature_fun=get_tfidf_features, ngrams=(min_r, max_r))
        results[min_r-1, max_r-1] = mean(accuracies)
        print('Accuracy for ngrams range ({}, {}): {}'.format(min_r, max_r, mean(accuracies)))
fig, ax = plt.subplots(figsize=(4,4))
sns.heatmap(results, annot=True, fmt='.4g', xticklabels=[1,2,3,4], yticklabels=[1,2,3,4],
            cmap='Blues')
plt.ylabel('Ngram min')
plt.xlabel('Ngram max')
plt.show()<jupyter_output><empty_output><jupyter_text>Based on this, we will choose the range (2, 4). Let's calculate the features for our training set and look at the most frequent n-grams:<jupyter_code>count_vectorizer, _, tfidf_features = get_tfidf_features(train_dataset['string'], ngrams=(2, 4))
tfidf_feature_names = count_vectorizer.get_feature_names()
print('Number of features: {}'.format(len(tfidf_feature_names)))

occurrences = count_vectorizer.fit_transform(train_dataset['string']).toarray().sum(axis=0)
indices = np.argsort(occurrences)[::-1]
tfidf_feature_names = np.array(tfidf_feature_names)[indices]
occurrences = np.array(occurrences)[indices]
print('\n'.join(['{}: {}'.format(p[0], p[1])
                 for p in list(zip(tfidf_feature_names, occurrences))[:50]]))<jupyter_output>Number of features: 23956
capword capword: 195164
uclett dot: 149612
lcword lcword: 137550
capword comma: 123393
capword lcword: 117591
lcword capword: 107489
dot capword: 104056
lcword lcword lcword: 86565
comma uclett: 79783
capword lcword capword: 77247
comma uclett dot: 77209
comma capword: 67107
capword comma uclett: 66162
capword comma uclett dot: 65498
lcword lcword lcword lcword: 63583
capword capword capword: 59982
capword dot: 57306
lcword capword capword: 53554
uclett dot capword: 49383
capword capword lcword: 47637
num dash: 43067
dash num: 41997
dot uclett: 41868
capword lcword capword capword: 40854
dot uclett dot: 39433
num dash num: 39354
uclett dot uclett: 39183
uclett dot uclett dot: 38238
dot capword comma: 38229
capword capword lcword capword: 37645
uclett dot capword comma: 35182
lcword dot: 33125
comma num: 32657
dot end: 31561
num dot: 30467
start capword: 30378
capword capword dot: 29627
year rpar: 29383
num lpar: 29117
num comma: 28748
lpar year: 28330
lpar yea[...]<jupyter_text>We can see all types of ngrams. In most cases we can guess where they came from, eg. "lcword lcword lcword lcword" is typically a fragment of the title, and "uclett dot capword comma" looks like a fragment from the authors' list.We can also check which features are the most correlated with the styles. We will use chi-squared statistic for this:<jupyter_code>for style in STYLES:
    print()
    print('Style: {}'.format(style))
    features_chi2 = chi2(tfidf_features, train_dataset['style'] == style)[0]
    indices = np.argsort(features_chi2)
    feature_names = np.array(count_vectorizer.get_feature_names())[indices]
    bigrams = [v for v in feature_names if len(v.split(' '))  == 2]
    trigrams = [v for v in feature_names if len(v.split(' '))  == 3]
    fourgrams = [v for v in feature_names if len(v.split(' '))  == 4]
    print('Most correlated bigrams: {}'.format('|'.join(bigrams[-5:])))
    print('Most correlated trigrams: {}'.format('|'.join(trigrams[-5:])))
    print('Most correlated fourgrams: {}'.format('|'.join(fourgrams[-5:])))<jupyter_output>Style: acm-sig-proceedings
Most correlated bigrams: dot lcword|rbracket year|dot year|rpar comma|rbracket capword
Most correlated trigrams: uclett dot year|num rbracket capword|rpar comma num|rbracket capword comma|year rpar comma
Most correlated fourgrams: num rbracket capword comma|dot year dot capword|rbracket capword comma uclett|lpar year rpar comma|year rpar comma num

Style: american-chemical-society
Most correlated bigrams: year comma|capword year|start lpar|semicolon capword|dot semicolon
Most correlated trigrams: start lpar num|comma num comma|semicolon capword comma|dot semicolon capword|uclett dot semicolon
Most correlated fourgrams: comma uclett dot semicolon|year comma num comma|dot semicolon capword comma|semicolon capword comma uclett|uclett dot semicolon capword

Style: american-chemical-society-with-titles
Most correlated bigrams: year comma|capword year|start lpar|semicolon capword|dot semicolon
Most correlated trigrams: start lpar num|comma num comma|semicolon capw[...]<jupyter_text>Feature selectionSo far we used all n-grams present in the training set. Some of them are very rare and don't provide a lot of information for the training algorithm. Let's try using only n most frequent ngrams for different values of n, and see how it affects the accuracy.<jupyter_code>nfeatures_results = []
nfeatures_values = range(100, 5100, 100)
for n in nfeatures_values:
    accuracies, _ = evaluate_cv(train_dataset, LogisticRegression(random_state=0),
                                feature_fun=get_tfidf_features, nfeatures=n, ngrams=(2, 4))
    nfeatures_results.append(mean(accuracies))
    print('Accuracy for {} features: {}'.format(n, mean(accuracies)))
plt.plot(nfeatures_values, nfeatures_results, color='#4f5858')
plt.xticks(list(range(200, 5100, 600)))
plt.xlabel('Number of features')
plt.ylabel('Accuracy')
plt.show()<jupyter_output><empty_output><jupyter_text>It seems that the more features we have the higher accuracy.Choosing the most frequent features might not be optimal. Some frequent features will be useless because they don't provide any information about the target style. We can perform feature selection to choose a number of most useful features.By passing *feature_selector* to the evaluation function we make sure the best features are selected in the training phase. In this case, the selection is done using feature importance scores from Random Forest.Similarly as before, we will test this with varying number of chosen features:<jupyter_code>nfeatures_results_rf = []
nfeatures_values = range(100, 5100, 100)
for n in nfeatures_values:
    accuracies, _ = evaluate_cv(train_dataset, LogisticRegression(random_state=0),
                                feature_fun=get_tfidf_features,
                                feature_selector=select_features_rf,
                                nfeatures=n, ngrams=(2, 4))
    nfeatures_results_rf.append(mean(accuracies))
    print('Accuracy for {} features: {}'.format(n, mean(accuracies)))
plt.plot(nfeatures_values, nfeatures_results, color='#4f5858')
plt.plot(nfeatures_values, nfeatures_results_rf, color='#3eb1c8')
plt.legend(['without selection', 'with selection (RF)'], bbox_to_anchor=(0.9, 1.2), ncol=3)
plt.xticks(list(range(200, 5100, 600)))
plt.xlabel('Number of features')
plt.ylabel('Accuracy')
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that in a small range RF selection results in slightly better performance, but as we increase the number of features, no selection outperforms RF-based selection.We will also try a simpler selection method, based on chi-squared correlations:<jupyter_code>nfeatures_results_chi2 = []
nfeatures_values = range(100, 5100, 100)
for n in nfeatures_values:
    accuracies, _ = evaluate_cv(train_dataset, LogisticRegression(random_state=0),
                                feature_fun=get_tfidf_features,
                                feature_selector=select_features_chi2,
                                nfeatures=n, ngrams=(2, 4))
    nfeatures_results_chi2.append(mean(accuracies))
    print('Accuracy for {} features: {}'.format(n, mean(accuracies)))
plt.plot(nfeatures_values, nfeatures_results, color='#4f5858')
plt.plot(nfeatures_values, nfeatures_results_rf, color='#3eb1c8')
plt.plot(nfeatures_values, nfeatures_results_chi2, color='#ef3340')
plt.legend(['without selection', 'with selection (RF)', 'with selection (Chi^2)'],
           bbox_to_anchor=(1.1, 1.2), ncol=3)
plt.xticks(list(range(200, 5100, 600)))
plt.xlabel('Number of features')
plt.ylabel('Accuracy')
plt.show()<jupyter_output><empty_output><jupyter_text>The best accuracy was given by chi-squared selection with 5000 features. We will use that from now on. Final model<jupyter_code>accuracies, _ = evaluate_cv(train_dataset, LogisticRegression(random_state=0),
                            feature_fun=get_tfidf_features, feature_selector=select_features_chi2,
                            nfeatures=5000, ngrams=(2, 4))
print('Fold accuracies: {}'.format(accuracies))
print('Mean accuracy: {}'.format(mean(accuracies)))<jupyter_output>Fold accuracies: [0.9400484457004441, 0.9454752772882762, 0.9391561353444762, 0.9412355600200905, 0.940557705666369]
Mean accuracy: 0.9412946248039312<jupyter_text>Let's also add one more feature (string length) to our feature representation.<jupyter_code>accuracies, dataset_results_cv = evaluate_cv(train_dataset, LogisticRegression(random_state=0),
                                             feature_fun=get_features,
                                             feature_selector=select_features_chi2,
                                             nfeatures=5000, ngrams=(2, 4))
print('Fold accuracies: {}'.format(accuracies))
print('Mean accuracy: {}'.format(mean(accuracies)))<jupyter_output>Fold accuracies: [0.9404521598708114, 0.9465118689748109, 0.9390542193232776, 0.9403314917127071, 0.9409546492011511]
Mean accuracy: 0.9414608778165516<jupyter_text>Let's see the confusion matrix:<jupyter_code>matrix = confusion_matrix(dataset_results_cv['style_true'], dataset_results_cv['style_pred'])

fig, ax = plt.subplots(figsize=(10,10))
sns.heatmap(matrix, annot=True, fmt='d', xticklabels=STYLES, yticklabels=STYLES + ['unknown'],
            cmap='Blues')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()<jupyter_output><empty_output><jupyter_text>Everything looks pretty good apart from the pair (chicago-author-date, american-sociological-association), which seem to be confused a lot by the classifier.This function will show the reference string pairs from the two given  styles for all DOI that were mislabelled.<jupyter_code>def get_diff(actual, predicted):
    diff_df = dataset_results_cv.loc[dataset_results_cv['style_true'] == actual]
    diff_dois = diff_df.loc[diff_df['style_pred'] == predicted]['doi']
    diff_df = dataset_results_cv.loc[dataset_results_cv['doi'].isin(diff_dois)]
    diff_df = diff_df.loc[diff_df['style_true'].isin([actual, predicted])].sort_values(by='doi')
    strings = [re.sub('^\(1\)', '', re.sub('^\[1\]', '',
                                           re.sub('^1', '', re.sub('\.$', '', s)))).strip()
               for s in diff_df['string']]
    strings_0 = [s for i, s in enumerate(strings) if i%2 == 0]
    strings_1 = [s for i, s in enumerate(strings) if i%2 == 1]
    return list(zip(strings_0, strings_1))
get_diff('chicago-author-date', 'american-sociological-association')<jupyter_output><empty_output><jupyter_text>In a lot of cases the reference strings are very similar, or identical. No wonder the classifier has problems differentiating between the styles.Finally, let's test a few learning algorithms using previously chosen parameters:<jupyter_code>algo_results = []
for name, algo in [('Naive Bayes', MultinomialNB()),
                   ('Logistic Regression', LogisticRegression(random_state=0)),
                   ('Linear SVC', LinearSVC(random_state=0)),
                   ('Random Forest', RandomForestClassifier(n_estimators=200, max_depth=3,
                                                            random_state=0))]:
    accuracies, _ = evaluate_cv(train_dataset, algo, feature_fun=get_features,
                                feature_selector=select_features_chi2,
                                nfeatures=5000, ngrams=(2, 4))
    algo_results.extend([(name, fold, accuracy) for fold, accuracy in enumerate(accuracies)])
    print('Accuracy for {} algorithm {}'.format(name, mean(accuracies)))
cv_df = pd.DataFrame(algo_results, columns=['model', 'fold', 'accuracy'])
ax = sns.boxplot(x='model', y='accuracy', data=cv_df, color="#d8d2c4")
ax.set_xlabel('')
plt.setp(ax.get_xticklabels(), rotation=45)
plt.show()<jupyter_output><empty_output><jupyter_text>It seems logistic regression results in the best accuracy averaged over the folds. Also the variance of the results in this case is small.Finally, let's fit the model to the entire train set and evaluate on the main test set:<jupyter_code>count_vectorizer, tfidf_transformer, train_features = \
    get_features(train_dataset['string'], nfeatures=5000, feature_selector=select_features_chi2,
                 ngrams=(2, 4))
_, _, test_features = get_features(test_dataset['string'], count_vectorizer=count_vectorizer,
                                   tfidf_transformer=tfidf_transformer)
model = LogisticRegression(random_state=0).fit(train_features, train_dataset['style'])
prediction = model.predict(test_features)
print('The final accuracy: {}'.format(accuracy_score(test_dataset['style'], prediction)))<jupyter_output>The final accuracy: 0.941838903713084
{'fraction_non_alphanumeric': 0.09805392731535756, 'fraction_numerical': 0.042438452520515826, 'mean_word_length': 5.027699265121537, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 96, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '603221', 'license': 'MIT', 'n_tokens_mistral': 7163, 'n_tokens_neox': 6316, 'n_words': 2016, 'path': 'analyses/citation_style_classification.ipynb', 'repo_name': 'CrossRef/citation-style-classifier', 'size': '21452'}