<commit_before># dfr-analysis

Bits and pieces to help analyze the data available from JSTOR's Data for Research (DfR) service. Skews to my needs as a literary scholar and digital-humanist-in-progress. Skews to my amateurishness as a programmer. Very much in-progress, hacked together, catch-as-catch-can, I am not an expert, I am not a lawyer, etc., etc., etc. Use and share freely, at your own risk. 

## metadata.R

Functions for manipulating metadata from DfR with R.

## topics.R

Basic R script for loading information from a topic model created by MALLET, getting basic information from it, and making time-series plots of various kinds. In progress.

## topics_rmallet.R

Thanks to the advent of [the mallet interface for R](http://www.cs.princeton.edu/~mimno/R/), it's possible to avoid having to do the kinds of reprocessing of the output from the mallet command-line tool that I used to need to do (using the perl scripts listed below). In fact it's possible to run mallet itself from within R. This script provides basic functions for using mallet to topic-model wordcount data from DfR. I use it to put together a data frame with document metadata and document-topic proportions, which can then work with the same plotting functions in topics.R. But much more is possible, given access, within R, to the mallet topic model object itself.

## Legacy Scripts

### combine_citations

A simple shell script to combine multiple `citations.CSV` metadata files
from jstor and get rid of any duplicate lines. Usage:

    combine_citations citations1.CSV citations2.CSV ... > citations_all.csv

Sanity-check results before using.
<commit_msg>readme, now with backwards-incompatibility warning<commit_after># dfr-analysis

Bits and pieces to help analyze the data available from JSTOR's Data for Research (DfR) service. Skews to my needs as a literary scholar and digital-humanist-in-progress. Skews to my amateurishness as a programmer. Very much in-progress, hacked together, catch-as-catch-can, I am not an expert, I am not a lawyer, etc., etc., etc. Use and share freely, at your own risk. 

## metadata.R

Functions for manipulating metadata from DfR with R.

## topics.R

Basic R script for loading information from a topic model created by MALLET, getting basic information from it, and making time-series plots of various kinds. In progress.

*Note.* These functions were written to use in conjunction with the "legacy scripts" below. I am moving to the R interface for mallet (see below) and so am using the data-reading functions in this file less and less. Since I gather a few people have been making use of some of this code, I have left the legacy functions in. But this is really for my own use, so I won't maintain backwards compatibility longer than I feel like it.

## topics_rmallet.R

Thanks to the advent of [the mallet interface for R](http://www.cs.princeton.edu/~mimno/R/), it's possible to avoid having to do the kinds of reprocessing of the output from the mallet command-line tool that I used to need to do (using the perl scripts listed below). In fact it's possible to run mallet itself from within R. This script provides basic functions for using mallet to topic-model wordcount data from DfR. I use it to put together a data frame with document metadata and document-topic proportions, which can then work with the same plotting functions in topics.R. But much more is possible, given access, within R, to the mallet topic model object itself.

## Legacy scripts

### combine_citations

A simple shell script to combine multiple `citations.CSV` metadata files
from jstor and get rid of any duplicate lines. Usage:

    combine_citations citations1.CSV citations2.CSV ... > citations_all.csv

Sanity-check results before using.

{'fraction_non_alphanumeric': 0.05555555555555555, 'fraction_numerical': 0.001063264221158958, 'mean_word_length': 4.233657858136301, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3764522', 'n_tokens_mistral': 1002, 'n_tokens_neox': 950, 'n_words': 564}