<commit_before># dfr-analysis

Bits and pieces to help analyze the data available from JSTOR's Data for Research (DfR) service. Skews to my needs as a literary scholar and digital-humanist-in-progress. Skews to my amateurishness as a programmer. Very much in-progress, hacked together, catch-as-catch-can, I am not an expert, I am not a lawyer, etc., etc., etc. Use and share freely, at your own risk. 

## metadata.R

Functions for manipulating metadata from DfR with R. The core function is `read_metadata()` for reading and combining `citations.CSV` files. I make particularly frequent use of `pubdate_Date()`, which converts the "pubdate" field to `Date` objects. For browsing, `cite_articles()` turns rows of metadata into a human-readable citation.


## topics_rmallet.R

Thanks to the advent of [the mallet interface for R](http://www.cs.princeton.edu/~mimno/R/), it's possible to avoid having to do the kinds of reprocessing of the output from the mallet command-line tool that I used to need to do (using the perl scripts listed below). In fact it's possible to run mallet itself from within R. This script provides basic functions for using mallet to topic-model wordcount data from DfR, and then to read and write the results of that output, as well as to tally it up by year. The functions in this file are explained in comments.

I also provide some functions for reading mallet's "instances" from within R, so that you can recover the documents you fed to the topic modeler (in post-processed form, i.e. with stopwords removed, etc.).

I have not been parsimonious about using other R packages, so there are a lot of dependencies here.

### Basic example usage within R

```R
# let's imagine you have this repository in ~/projects/dfr-analysis
# Download and unzip DFR data to, say ~/dfr/test/
# you get metadata in ~/dfr/test/citations.CSV
# and wordcounts per document in ~/dfr/test/wordcounts/*.CSV

setwd("~/projects/dfr-analysis")
source("topics_rmallet.R")
topics_rmallet_setup()    # loads metadata.R, rJava, other dependencies
m <- model_documents(citations.file="~/dfr/test/citations.CSV",
    dirs="~/dfr/test/wordcounts/",stoplist.file="stoplist/long.txt",
    num.topics=12)

source("topics_vis.R")
tm_wide <- doc_topics_wide(m$doc_topics,m$metadata)
tm_yearly_line_plot(tm_wide=tm_wide,facet=T)
```

## topics_vis.R

This script provides some `ggplot2` visualizations that make use of various aspects of the mallet output, with an emphasis on following topics over time. The `topic_keyword_plot()` gives a way of looking at "top key words" for a topic that preserves information about their weights. `topic_report()` generates a folder of images (multiple plots each) giving an overview of each topic in the model.

There are also some basic visualizations of the words in the corpus, which need various slices of the term-document matrix and the term-document-topic array (computed by functions in `topics_rmallet.R`).

## A note on licensing
<commit_msg>move example up; note about demo<commit_after># dfr-analysis

Bits and pieces to help analyze the data available from JSTOR's Data for Research (DfR) service. Skews to my needs as a literary scholar and digital-humanist-in-progress. Skews to my amateurishness as a programmer. Very much in-progress, hacked together, catch-as-catch-can, I am not an expert, I am not a lawyer, etc., etc., etc. Use and share freely, at your own risk. 

## Basic example usage within R

```R
# let's imagine you have this repository in ~/projects/dfr-analysis
# Download and unzip DFR data to, say ~/dfr/test/
# you get metadata in ~/dfr/test/citations.CSV
# and wordcounts per document in ~/dfr/test/wordcounts/*.CSV

setwd("~/projects/dfr-analysis")
source("topics_rmallet.R")
topics_rmallet_setup()    # loads metadata.R, rJava, other dependencies
m <- model_documents(citations.file="~/dfr/test/citations.CSV",
    dirs="~/dfr/test/wordcounts/",stoplist.file="stoplist/long.txt",
    num.topics=12)

source("topics_vis.R")
tm_wide <- doc_topics_wide(m$doc_topics,m$metadata)
tm_yearly_line_plot(tm_wide=tm_wide,facet=T)
```

A slightly fuller demonstration can be found in the `demo/` subdirectory here.

## A guide to the files

### metadata.R

Functions for manipulating metadata from DfR with R. The core function is `read_metadata()` for reading and combining `citations.CSV` files. I make particularly frequent use of `pubdate_Date()`, which converts the "pubdate" field to `Date` objects. For browsing, `cite_articles()` turns rows of metadata into a human-readable citation.


### topics_rmallet.R

Thanks to the advent of [the mallet interface for R](http://www.cs.princeton.edu/~mimno/R/), it's possible to avoid having to do the kinds of reprocessing of the output from the mallet command-line tool that I used to need to do (using the perl scripts listed below). In fact it's possible to run mallet itself from within R. This script provides basic functions for using mallet to topic-model wordcount data from DfR, and then to read and write the results of that output, as well as to tally it up by year. The functions in this file are explained in comments.

I also provide some functions for reading mallet's "instances" from within R, so that you can recover the documents you fed to the topic modeler (in post-processed form, i.e. with stopwords removed, etc.).

I have not been parsimonious about using other R packages, so there are a lot of dependencies here.

### topics_vis.R

This script provides some `ggplot2` visualizations that make use of various aspects of the mallet output, with an emphasis on following topics over time. The `topic_keyword_plot()` gives a way of looking at "top key words" for a topic that preserves information about their weights. `topic_report()` generates a folder of images (multiple plots each) giving an overview of each topic in the model.

There are also some basic visualizations of the words in the corpus, which need various slices of the term-document matrix and the term-document-topic array (computed by functions in `topics_rmallet.R`).

## A note on licensing

{'fraction_non_alphanumeric': 0.08026920551543007, 'fraction_numerical': 0.000984898227183191, 'mean_word_length': 4.600183823529412, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '844069', 'n_tokens_mistral': 1784, 'n_tokens_neox': 1698, 'n_words': 824}