<commit_before>Classics-Research
=================



<commit_msg>updated read me<commit_after>Classics-Research
=================

This is code written by <NAME> and <NAME> for performing similarity analysis on Latin tragedies written by the Roman statesman Seneca the Younger. Essentially, the central idea of this work is to provide quantitative analysis for an authorship attribution problem revolving around Seneca. Two of Seneca's plays "Octavia" and "<NAME>" are disputed as having been written by Seneca. 

## Features

Besides standard inference algorithms that leverage the results of machine learning, we also implement several key text-representation methods. In particular, we implement:

- Term-frequency inverse document frequency
- Functional n-gram probabilities

## Demo

```
from os import listdir
from os.path import isfile, join

from Utilities.Text import Text
from Utilities.FunctionalNGram import FunctionalNGram as FNG
from Utilities.FunctionalNGram import combine_n_gram_features as combine

from sklearn import svm
from pprint import pprint
import numpy as np

from sklearn.decomposition import PCA

def main():
	path = "./Texts/"
	files = [f for f in listdir(path) if isfile(join(path, f))]
	pprint(files)

	# Extract n gram probability features from the texts and 
	# construct them as a numpy 2D array. We also returns the 
	# n grams under consideration in the experiments here, 
	# although we have no particular use for them.
	features, n_grams = combine([FNG(Text(path + f)) for f in files])

	apply_pca = False

	if apply_pca:
		pca = PCA(n_components = features.shape[1])
		x = {
			"train" : pca.fit_transform(features[[0, 2, 4, 5, 6, 7], :]),
			"test" : pca.transform(features[[1, 3], :])
			}
	else:
		x = {
			"train" : features[[0, 2, 4, 5, 6, 7], :],
			"test" : features[[1, 3], :]
			}

	
	# Unfortunately, it does not appear to be possible to derive a perfect
	# accuracy solution in the grid search specified below. However, it is
	# provided here anyway for educational purposes.
	grid_search = False

	if grid_search:
		for kernel in ["rbf", "linear", "sigmoid", "poly"]:
			for nu in np.linspace(0.001,1.0,200):
				for gamma in np.linspace(0.0,10.0,200):

					clf = svm.OneClassSVM(nu = nu, kernel = kernel, gamma = gamma)
					clf.fit(x["train"])

					y = {
						"train" : clf.predict(x["train"]),
						"test" : clf.predict(x["test"])
						}
				

					if all(y["train"] == 1.0) and all(y["test"] == -1.0):
						pprint({"nu" : nu, "gamma" : gamma, "y" : y, "kernel" : kernel})


	# It is possible to achieve good results with nu = gamma = .1 and with
	# a radial basis function kernel.
	nu, kernel, gamma = 0.1, "rbf", 0.1
	clf = svm.OneClassSVM(nu = nu, kernel = kernel, gamma = gamma)
	clf.fit(x["train"])

	y = {
		"train" : clf.predict(x["train"]),
		"test" : clf.predict(x["test"])
	}

	metrics = {
		"train" : clf.decision_function(x["train"]),
		"test" : clf.decision_function(x["test"])
	}

	pprint({"nu" : nu, "gamma" : gamma, "y" : y, "kernel" : kernel, "metrics" : metrics})


if __name__ == "__main__":
	main()
```

## Dependencies

* [Numpy](http://www.numpy.org/) Standard numerical computations with vectors
* [Scipy](http://www.scipy.org/) Used for calculations involving statistical distributions
* [scikit-learn](http://scikit-learn.org/stable/) Python machine learning library



{'fraction_non_alphanumeric': 0.1361740166865316, 'fraction_numerical': 0.013110846245530394, 'mean_word_length': 3.973333333333333, 'pattern_counts': {'":': 1, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3325958', 'n_tokens_mistral': 1216, 'n_tokens_neox': 1070, 'n_words': 372}