<jupyter_start><jupyter_text>Ružička: Authorship Verification in Python In this notebook, we offer a quick tutorial as to how you could use the code in this repository. While the package is very much geared towards our own work in authorship verification, you might some of the more general functions useful. All feedback and comments are welcome. This code assumes Python 2.7+ (Python 3 has not been tested). You do not need to install the library to run the code below, but please note that there are a number of well-known third-party Python libraries, including:+ numpy+ scipy+ scikit-learn+ matplotlib+ seaborn+ numbaand preferably (for GPU acceleration and/or JIT-compilation):+ theano+ numbaproWe recommend installing Continuum's excellent [Anaconda Python framework](https://www.continuum.io/downloads), which comes bundled with most of these dependencies.  Walk through By default, we assume that your data sets are stored in a directory the format on the PAN 2014 track on authorship attribution: a directory should minimally include one folder per verification problem (an `unknown.txt` and at least one `known01.txt`) and a `truth.txt`. E.g. for the corpus of Dutch essays (`../data/2014/du_essays/train`), `truth.txt` contains has a tab-separated line with the ground truth for each problem:```DE001 YDE002 YDE003 NDE004 NDE005 NDE006 NDE007 NDE008 Y...``` To inspect the problems:<jupyter_code>ls ../data/2014/du_essays/train<jupyter_output>[34mDE001[m[m/         [34mDE014[m[m/         [34mDE027[m[m/         [34mDE040[m[m/         [34mDE053[m[m/         [34mDE066[m[m/         [34mDE079[m[m/         [34mDE092[m[m/
[34mDE002[m[m/         [34mDE015[m[m/         [34mDE028[m[m/         [34mDE041[m[m/         [34mDE054[m[m/         [34mDE067[m[m/         [34mDE080[m[m/         [34mDE093[m[m/
[34mDE003[m[m/         [34mDE016[m[m/         [34mDE029[m[m/         [34mDE042[m[m/         [34mDE055[m[m/         [34mDE068[m[m/         [34mDE081[m[m/         [34mDE094[m[m/
[34mDE004[m[m/         [34mDE017[m[m/         [34mDE030[m[m/         [34mDE043[m[m/         [34mDE056[m[m/         [34mDE069[m[m/         [34mDE082[m[m/         [34mDE095[m[m/
[34mDE005[m[m/         [34mDE018[m[m/         [34mDE031[m[m/         [34mDE044[m[m/         [34mDE057[m[m/         [34mDE070[m[m/         [34mDE083[m[m/         [34mDE096[m[...]<jupyter_text>Let us now load the set of development problems for the Dutch essays:<jupyter_code>from ruzicka.utilities import *
D = '../data/2014/du_essays/'
dev_train_data, dev_test_data = load_pan_dataset(D+'train')<jupyter_output><empty_output><jupyter_text>This functions loads all documents and splits the development data into a development part (the known documents) and a testing part (the unknown documents). We can unpack these as follows:<jupyter_code>dev_train_labels, dev_train_documents = zip(*dev_train_data)
dev_test_labels, dev_test_documents = zip(*dev_test_data)<jupyter_output><empty_output><jupyter_text>Let us have a look at the actual test texts:<jupyter_code>from __future__ import print_function
for doc in dev_test_documents[:10]:
    print('+ ', doc[:70])<jupyter_output>+  ﻿Dankzij het internet zijn we een grote bron aan informatie rijker .
+  ﻿Het is dus begrijpelijk dat de commerciële zenders meer reclame mo
+  ﻿" Hey , vuile nicht ! Hangt er nog stront aan je lul ? " . Dergelij
+  ﻿Gelijkheid tussen man en vrouw is iets dat ons al eeuwen in de ban 
+  ﻿Gisteren was er opnieuw een protest tegen homofilie in de grootstad
+  ﻿Voetbal is vandaag de dag zonder twijfel de populairste sport in Be
+  ﻿Door de ongekende groei van nieuwsbronnen en de opkomst van het int
+  ﻿Woordenboekgebruik uit interesse De categorie woordenboekgebruikers
+  ﻿Ze bouwden een tegencultuur op die alles verwierp waar hun ouders a
+  ﻿Als we hier in België op straat rondlopen , merken we dat er zeer<jupyter_text>For each of these documents we need to decide whether or not they were in fact written by the target authors proposed:<jupyter_code>for doc in dev_test_labels[:10]:
    print('+ ', doc[:70])<jupyter_output>+  DE001
+  DE002
+  DE003
+  DE004
+  DE005
+  DE006
+  DE007
+  DE008
+  DE009
+  DE010<jupyter_text>The first and crucial step is to vectorize the documents using a vector space model. Below, we use generic example, using the 10,000 most common word unigrams and a plain *tf* model:<jupyter_code>from ruzicka.vectorization import Vectorizer
vectorizer = Vectorizer(mfi = 10000,
                        vector_space = 'tf',
                        ngram_type = 'word',
                        ngram_size = 1)

dev_train_X = vectorizer.fit_transform(dev_train_documents).toarray()
dev_test_X = vectorizer.transform(dev_test_documents).toarray()<jupyter_output><empty_output><jupyter_text>Note that we use `sklearn` conventions here: we fit the vectorizer only on the vocabulary of the known documents and apply it it later to the unknown documents (since in real life too, we will not necessarily know the known documents in advance). This gives us two compatible corpus matrices:<jupyter_code>print(dev_train_X.shape)
print(dev_test_X.shape)<jupyter_output>(172, 9977)
(96, 9977)<jupyter_text>We now encode the author labels in the development problem sets as integers, using sklearn's convenient `LabelEncoder`:<jupyter_code>from sklearn.preprocessing import LabelEncoder
label_encoder = LabelEncoder()
label_encoder.fit(dev_train_labels + dev_test_labels)
dev_train_y = label_encoder.transform(dev_train_labels)
dev_test_y = label_encoder.transform(dev_test_labels)
print(dev_test_y)<jupyter_output>[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95]<jupyter_text>We now construct and fit an 'O2' verifier: this extrinsic verification technique is based on the General Imposters framework. We apply it with the minmax metric and a profile base, meaning that the known documents for each author will be represented as a mean centroid:<jupyter_code>from ruzicka.Order2Verifier import Order2Verifier
dev_verifier = Order2Verifier(metric = 'minmax',
                              base = 'profile',
                              nb_bootstrap_iter=100,
                              rnd_prop = 0.5)
dev_verifier.fit(dev_train_X, dev_train_y)<jupyter_output><empty_output><jupyter_text>We can now obtain the probability which this O1 verifier would assign to each combination of an unknown document and the target author suggested in the problem:<jupyter_code>dev_test_scores = dev_verifier.predict_proba(test_X = dev_test_X,
                                             test_y = dev_test_y,
                                             nb_imposters = 30)<jupyter_output>ruzicka/Order2Verifier.py:191: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if rnd_feature_idxs == 'all': # use entire feature space
ruzicka/Order2Verifier.py:252: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if rnd_feature_idxs == 'all':<jupyter_text>This gives us as an array of probability scores for each problem, corresponding to the number of iterations in which the target's author's profile was closer to the anonymous document than to one of the imposters:<jupyter_code>print(dev_test_scores)<jupyter_output>[ 0.69        0.61000001  0.          0.          0.08        0.07        0.
  1.          1.          0.75999999  0.49000001  0.31        0.94        0.94
  0.01        0.38999999  0.54000002  0.          0.03        0.36000001
  0.          0.          0.56        0.38999999  0.          0.81999999
  0.          0.52999997  0.04        0.          0.          0.01
  0.25999999  0.          0.02        0.18000001  0.          0.07        0.09
  0.          0.23        0.70999998  0.02        0.77999997  1.          0.
  0.38        0.01        0.          0.23999999  0.01        0.40000001
  0.03        0.38        0.72000003  0.          0.02        0.76999998
  0.02        0.83999997  0.98000002  0.64999998  0.97000003  0.50999999
  0.68000001  0.89999998  0.41999999  0.16        0.56        0.87
  0.34999999  0.01        0.02        0.50999999  0.07        0.12
  0.20999999  0.          0.99000001  0.          0.88        0.38        0.
  0.          1.          0.          1.     [...]<jupyter_text>Let us now load the ground truth to check how well we did:<jupyter_code>dev_gt_scores = load_ground_truth(
                    filepath=os.sep.join((D, 'train', 'truth.txt')),
                    labels=dev_test_labels)
print(dev_gt_scores)<jupyter_output>[1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0]<jupyter_text>There is one final step needed: the PAN evaluation measures allow systems to leave a number of difficult problems unanswered, by setting the probability exactly at 0.5. To account for this strict threshold, we fit a score shifter, which will attempt to rectify mid-range score to 0.5. We can tune these parameters as follows:<jupyter_code>from ruzicka.score_shifting import ScoreShifter
shifter = ScoreShifter()
shifter.fit(predicted_scores=dev_test_scores,
            ground_truth_scores=dev_gt_scores)
dev_test_scores = shifter.transform(dev_test_scores)<jupyter_output>p1 for optimal combo: 0.08
p2 for optimal combo: 0.35
AUC for optimal combo: 0.955729166667
c@1 for optimal combo: 0.943142361111<jupyter_text>As you can see, this shifter optimizes 2 parameters using a grid search: all values in between *p1* and *p2* will be rectified to 0.5:<jupyter_code>print(dev_test_scores)<jupyter_output>[0.79849999845027919, 0.7465000092983245, 0.0, 0.0, 0.0063999998569488539, 0.005600000023841859, 0.0, 1.0, 1.0, 0.8439999938011169, 0.66850000619888306, 0.5, 0.96099999845027928, 0.96099999845027928, 0.00079999998211860673, 0.60349999070167537, 0.70100001394748679, 0.0, 0.00239999994635582, 0.58400000929832463, 0.0, 0.0, 0.71400000154972076, 0.60349999070167537, 0.0, 0.88299999535083762, 0.0, 0.69449998140335079, 0.0031999999284744269, 0.0, 0.0, 0.00079999998211860673, 0.5, 0.0, 0.0015999999642372135, 0.5, 0.0, 0.005600000023841859, 0.5, 0.0, 0.5, 0.81149998605251317, 0.0015999999642372135, 0.85699998140335087, 1.0, 0.0, 0.59699999690055849, 0.00079999998211860673, 0.0, 0.5, 0.00079999998211860673, 0.61000000387430187, 0.00239999994635582, 0.59699999690055849, 0.81800001859664917, 0.0, 0.0015999999642372135, 0.85049998760223389, 0.0015999999642372135, 0.89599998295307159, 0.98700001239776602, 0.77249998450279234, 0.98050001859664904, 0.68149999380111692, 0.7920000046491622, 0.934999984[...]<jupyter_text>We can later apply this optimized score shifter to the test problems. Now the main question: how well would our O2 verifier perform on the development problems, given the optimal *p1* and *p2* found? We answer this question using the three evaluation measures used in the PAN competition.<jupyter_code>from ruzicka.evaluation import pan_metrics
dev_acc_score, dev_auc_score, dev_c_at_1_score = \
    pan_metrics(prediction_scores=dev_test_scores,
    ground_truth_scores=dev_gt_scores)
print('Accuracy: ', dev_acc_score)
print('AUC: ', dev_auc_score)
print('c@1: ', dev_c_at_1_score)
print('AUC x c@1: ', dev_auc_score * dev_c_at_1_score)<jupyter_output>Accuracy:  0.885416666667
AUC:  0.955729166667
c@1:  0.943142361111
AUC x c@1:  0.901388662833<jupyter_text>Our score shifting approach clearly pays off, since we are able to leave difficult problems unswered, yielding to a higher c@1 than pure accuracy. We can now proceed to the test problems. The following code block runs entire parallel to the approach above: only the score shifter isn't retrained again:<jupyter_code>train_data, test_data = load_pan_dataset(D+'test')
train_labels, train_documents = zip(*train_data)
test_labels, test_documents = zip(*test_data)
                
# vectorize:
vectorizer = Vectorizer(mfi = 10000,
                        vector_space = 'tf',
                        ngram_type = 'word',
                        ngram_size = 1)
train_X = vectorizer.fit_transform(train_documents).toarray()
test_X = vectorizer.transform(test_documents).toarray()
                
# encode author labels:
label_encoder = LabelEncoder()
label_encoder.fit(train_labels+test_labels)
train_y = label_encoder.transform(train_labels)
test_y = label_encoder.transform(test_labels)
                
# fit and predict a verifier on the test data:
test_verifier = Order2Verifier(metric = 'minmax',
                               base = 'profile',
                               nb_bootstrap_iter=100,
                               rnd_prop = 0.5)
test_verifier.fit(train_X, train_y)
test_scores = test_verifier.predict_proba(test_X=test_X,
                                          test_y=test_y,
                                          nb_imposters=30)
                
# load the ground truth:
test_gt_scores = load_ground_truth(
                    filepath=os.sep.join((D, 'test', 'truth.txt')),
                    labels=test_labels)
                
# apply the optimzed score shifter:
test_scores = shifter.transform(test_scores)
                
test_acc_score, test_auc_score, test_c_at_1_score = \
    pan_metrics(prediction_scores=test_scores,
                ground_truth_scores=test_gt_scores)

print('Accuracy: ', test_acc_score)
print('AUC: ', test_auc_score)
print('c@1: ', test_c_at_1_score)
print('AUC x c@1: ', test_auc_score * test_c_at_1_score)<jupyter_output>- # test documents processed: 10 out of 96
	 - # test documents processed: 20 out of 96
	 - # test documents processed: 30 out of 96
	 - # test documents processed: 40 out of 96
	 - # test documents processed: 50 out of 96
	 - # test documents processed: 60 out of 96
	 - # test documents processed: 70 out of 96
	 - # test documents processed: 80 out of 96
	 - # test documents processed: 90 out of 96
Accuracy:  0.864583333333
AUC:  0.9609375
c@1:  0.911458333333
AUC x c@1:  0.875854492187<jupyter_text>While our final test results are a bit lower, the verifier seems to scale reasonably well to the unseen verification problems in the test set.  First Order Verification It is interesting now to compare the GI approach to a first-order verification system, which often yields very competitive results too. Our implementation closely resembles the system proposed by Potha and Stamatatos in 2014 (A Profile-based Method for Authorship Verification). We import and fit this O1 verifier:<jupyter_code>from ruzicka.Order1Verifier import Order1Verifier
dev_verifier = Order1Verifier(metric = 'minmax',
                              base = 'profile')
dev_verifier.fit(dev_train_X, dev_train_y)
dev_test_scores = dev_verifier.predict_proba(test_X = dev_test_X,
                                             test_y = dev_test_y)
print(dev_test_scores)<jupyter_output>[ 0.0508821   0.05295295 -0.05339944 -0.07909369 -0.02331865 -0.04220104
 -0.06020927  0.11833715  0.11711633  0.03420103  0.01194018 -0.00176835
  0.09044588  0.05795223 -0.10883117 -0.00071907 -0.08573282 -0.13027966
 -0.05026388 -0.01643515 -0.05558467 -0.12349176  0.0027076  -0.04140735
 -0.06439781 -0.01183951 -0.09243321 -0.03753805 -0.06817973 -0.10692203
 -0.08212757 -0.09001279 -0.06661606 -0.10339952 -0.09174156 -0.03461802
 -0.1220206  -0.05210984 -0.12378168 -0.08442163 -0.02438498  0.03309178
 -0.07402968  0.02882493  0.12914622 -0.14603448 -0.03053057 -0.05629373
 -0.10035634 -0.10980856 -0.07716274 -0.07025313 -0.0667429  -0.11839318
  0.02641141 -0.13112211 -0.03812957  0.05383098 -0.05459356  0.03681302
 -0.03131771  0.03050268  0.0914582   0.02064216  0.01521158  0.0497179
  0.00120807 -0.06035507  0.01666337  0.07360435 -0.15455794 -0.19472182
 -0.18665552 -0.02599692 -0.11922693 -0.1706109  -0.08144045 -0.09309399
  0.09763068 -0.08678317 -0.03580868 -0.03423667 -0.[...]<jupyter_text>Note that in this case, the 'probabilities' returned are only distance-based pseudo-probabilities and don't lie in the range of 0-1. Applying the score shifter is therefore quintessential with O1, since it will scale the distances to a more useful range:<jupyter_code>shifter = ScoreShifter()
shifter.fit(predicted_scores=dev_test_scores,
            ground_truth_scores=dev_gt_scores)
dev_test_scores = shifter.transform(dev_test_scores)
print(dev_test_scores)<jupyter_output>p1 for optimal combo: 0.4
p2 for optimal combo: 0.48
AUC for optimal combo: 0.900607638889
c@1 for optimal combo: 0.875217013889
[0.84273804257641682, 0.84579651967613234, 0.16055557333113935, 0.13136447581750335, 0.5, 0.5, 0.15281896211244364, 0.94236395287675823, 0.94056089246296626, 0.81810138143372435, 0.78522383099719839, 0.76497738691808459, 0.90117068640332054, 0.85318007055721723, 0.097579896593504079, 0.76652709278296061, 0.12382180468222422, 0.073212381489759837, 0.1641178680337276, 0.74331566101724755, 0.1580729506571803, 0.080924073032932753, 0.77158802155890516, 0.5, 0.14806038755174178, 0.75010306283465722, 0.11620952097510422, 0.5, 0.14376377501934579, 0.099748856395121779, 0.12791770548024634, 0.11895935299583767, 0.14554024993147938, 0.10375076667785683, 0.11699530335218733, 0.5, 0.082595451922209323, 0.16202068773225708, 0.080594699930370509, 0.1253114324598073, 0.5, 0.81646311591762899, 0.13711767047081092, 0.81016129564706429, 0.95832810646525068, 0.0553134182464505[...]<jupyter_text>And again, we are now ready to test the performance of O1 on the test problems.<jupyter_code>train_data, test_data = load_pan_dataset(D+'test')
train_labels, train_documents = zip(*train_data)
test_labels, test_documents = zip(*test_data)
                
# vectorize:
vectorizer = Vectorizer(mfi = 10000,
                        vector_space = 'tf',
                        ngram_type = 'word',
                        ngram_size = 1)
train_X = vectorizer.fit_transform(train_documents).toarray()
test_X = vectorizer.transform(test_documents).toarray()
                
# encode author labels:
label_encoder = LabelEncoder()
label_encoder.fit(train_labels+test_labels)
train_y = label_encoder.transform(train_labels)
test_y = label_encoder.transform(test_labels)
                
# fit and predict a verifier on the test data:
test_verifier = Order1Verifier(metric = 'minmax',
                               base = 'profile')
test_verifier.fit(train_X, train_y)
test_scores = test_verifier.predict_proba(test_X=test_X,
                                          test_y=test_y)
                
# load the ground truth:
test_gt_scores = load_ground_truth(
                    filepath=os.sep.join((D, 'test', 'truth.txt')),
                    labels=test_labels)
                
# apply the optimzed score shifter:
test_scores = shifter.transform(test_scores)
                
test_acc_score, test_auc_score, test_c_at_1_score = \
    pan_metrics(prediction_scores=test_scores,
                ground_truth_scores=test_gt_scores)

print('Accuracy: ', test_acc_score)
print('AUC: ', test_auc_score)
print('c@1: ', test_c_at_1_score)
print('AUC x c@1: ', test_auc_score * test_c_at_1_score)<jupyter_output>Accuracy:  0.5
AUC:  0.881944444444
c@1:  0.637478298611
AUC x c@1:  0.562220443914
{'fraction_non_alphanumeric': 0.09892927160307646, 'fraction_numerical': 0.18423566078520082, 'mean_word_length': 3.2672672672672673, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 62, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '435465', 'license': 'MIT', 'n_tokens_mistral': 9512, 'n_tokens_neox': 7041, 'n_words': 1894, 'path': 'code/Quickstart.ipynb', 'repo_name': 'mikekestemont/ruzicka', 'size': '19893'}