<commit_before>---
name: apis-text-mining-logs
layout: post
title: "text mining, apis, and parsing api logs"
date: 2019-03-21
author: <NAME>
sourceslug: _drafts/2019-03-21-apis-text-mining-logs.Rmd
tags:
- R
- API
- text-mining
- DOI
---



## Acquiring full text articles

[fulltext][] is an R package I maintain to obtain full text versions of research articles
for text mining.

It's a hard problem, with a spaghetti web of code. One of the hard problems is 
figuring out what the URL is for the full text version of an article. Publishers
do not have consistent URL patterns through time, and so you can not set rules once 
and never revisit them. 

The [Crossref API](https://github.com/CrossRef/rest-api-doc) has links available to 
full text versions for publishers that choose to share them. However, even if 
publishers choose to share their full text links, they may be out of date or completely
wrong (not actually lead to the full text). 

There's a variety of other APIs out there for getting links to articles, but none 
really hit the spot, which lead to the creation of the [ftdoi API][ftdoi]. 

## the ftdoi API

The [ftdoi API][ftdoi] is a web API, with it's main goal for getting a best guess at
the URL for the full text version of an article from it's DOI (this is done via the 
`/api/doi/{doi}/` route). The API gives back URLs for all those possible, including
pdf, xml, and html. Most publishers only give full text as PDF, but when XML is
available we give those URLs as well.

The API uses the rules maintained in the [pubpatterns](https://github.com/ropenscilabs/pubpatterns/tree/master/src)
repo. The rules are only rough guidelines though and often require at least one 
step of making a web request to the publishers site or another site, that's NOT 
specified in the pubpatterns rules. For example, the [Biorxiv file](https://github.com/ropenscilabs/pubpatterns/blob/master/src/biorxiv.json)
has notes about how to get the parts necessary for the full URL, but the actual logic 
to do so in in the API code base [here](https://github.com/ropenscilabs/pubpatternsapi/blob/master/utils.rb#L590-L601). 

The ftdoi API caches responses for requests for 24 hrs, so even if a request takes 5 seconds
or so to process, at least for the next 24 hrs it will be nearly instantaneous. We don't 
want to cache indefinitely because URLs may be changed at any time by the publishers.

The `fulltext` package uses the ftdoi API internally, mostly hidden from users, to 
get a full text URL.

## But why an API?

Why not just have a set of rules in the `fulltext` R package, and go from there?
An API was relatively easy for me to stand up, and i think it has many benefits:
can be used by anything/anyone, not just R users; updates to publisher specific
rules for generating URLs can evolve independently of `fulltext`; the logs
can be used as a tool to improve the API.

## what do people actually want?<commit_msg>It's -> its<commit_after>---
name: apis-text-mining-logs
layout: post
title: "text mining, apis, and parsing api logs"
date: 2019-03-21
author: <NAME>
sourceslug: _drafts/2019-03-21-apis-text-mining-logs.Rmd
tags:
- R
- API
- text-mining
- DOI
---



## Acquiring full text articles

[fulltext][] is an R package I maintain to obtain full text versions of research articles
for text mining.

It's a hard problem, with a spaghetti web of code. One of the hard problems is 
figuring out what the URL is for the full text version of an article. Publishers
do not have consistent URL patterns through time, and so you can not set rules once 
and never revisit them. 

The [Crossref API](https://github.com/CrossRef/rest-api-doc) has links available to 
full text versions for publishers that choose to share them. However, even if 
publishers choose to share their full text links, they may be out of date or completely
wrong (not actually lead to the full text). 

There's a variety of other APIs out there for getting links to articles, but none 
really hit the spot, which lead to the creation of the [ftdoi API][ftdoi]. 

## the ftdoi API

The [ftdoi API][ftdoi] is a web API, with it's main goal for getting a best guess at
the URL for the full text version of an article from its DOI (this is done via the 
`/api/doi/{doi}/` route). The API gives back URLs for all those possible, including
pdf, xml, and html. Most publishers only give full text as PDF, but when XML is
available we give those URLs as well.

The API uses the rules maintained in the [pubpatterns](https://github.com/ropenscilabs/pubpatterns/tree/master/src)
repo. The rules are only rough guidelines though and often require at least one 
step of making a web request to the publishers site or another site, that's NOT 
specified in the pubpatterns rules. For example, the [Biorxiv file](https://github.com/ropenscilabs/pubpatterns/blob/master/src/biorxiv.json)
has notes about how to get the parts necessary for the full URL, but the actual logic 
to do so in in the API code base [here](https://github.com/ropenscilabs/pubpatternsapi/blob/master/utils.rb#L590-L601). 

The ftdoi API caches responses for requests for 24 hrs, so even if a request takes 5 seconds
or so to process, at least for the next 24 hrs it will be nearly instantaneous. We don't 
want to cache indefinitely because URLs may be changed at any time by the publishers.

The `fulltext` package uses the ftdoi API internally, mostly hidden from users, to 
get a full text URL.

## But why an API?

Why not just have a set of rules in the `fulltext` R package, and go from there?
An API was relatively easy for me to stand up, and i think it has many benefits:
can be used by anything/anyone, not just R users; updates to publisher specific
rules for generating URLs can evolve independently of `fulltext`; the logs
can be used as a tool to improve the API.

## what do people actually want?
{'fraction_non_alphanumeric': 0.06347039754308138, 'fraction_numerical': 0.009213444804640846, 'mean_word_length': 4.0188356164383565, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '812284', 'n_tokens_mistral': 1755, 'n_tokens_neox': 1618, 'n_words': 926}