<commit_before>All of Plos (allofplos)
=======================

Copyright (c) 2017, Public Library of Science. MIT License, see
LICENSE.txt for more information.

Why allofplos?
--------------

This is for downloading/updating/maintaining a repository of all PLOS
XML article files. This can be used to have a copy of the PLOS text
corpus for further analysis. Use this program to download all PLOS XML
article files instead of doing web scraping.

Installation instructions
-------------------------

This program requires Python 3.4+.

Make a virtual environment:

``$ virtualenv allofplos``

Install Python dependencies inside the newly created virtual
environment:

``(allofplos)$ pip install -r requirements.txt``

How to run the program
----------------------

From inside the allofplos directory, run:

``(allofplos)$ python PLOS_Corpus.py``

The first time it runs it will download a >4.4 Gb zip file
(**AllofPLOS\_article\_XML.zip**) with all the XML files inside.
**Note**: Make sure that you have enough space in your device for the
zip file and for it content before running this command (at least 30Gb).
After this file is downloaded, it will extract it contents into
AllofPLOS\_article\_XML directory.

If you ever downloaded the corpus before, it will make an incremental
update to the existing corpus, the script checks for and then downloads
to a temporary folder:

-  individual new articles that have been published
-  of those new articles, checks whether they are corrections (and
   whether the linked corrected article has been updated)
-  checks whether there are VORs (Versions of Record) for uncorrected
   proofs in the main articles directory & downloads those
-  checks whether the newly downloaded articles are uncorrected proofs
   or not after all of these checks, it moves the new articles into the
   main articles folder.

Here’s what the print statements might look like on a typical run:

::

    147 new articles to download.
    147 new articles downloaded.
    3 corrected articles found.
    0 corrected articles downloaded with new xml.
    Creating new text list of uncorrected proofs from scratch.<commit_msg>Adding info to rst file<commit_after>All of Plos (allofplos)
=======================

Copyright (c) 2017, Public Library of Science. MIT License, see
LICENSE.txt for more information.

Why allofplos?
--------------

This is for downloading/updating/maintaining a repository of all PLOS
XML article files. This can be used to have a copy of the PLOS text
corpus for further analysis. Use this program to download all PLOS XML
article files instead of doing web scraping.

**NOTE**: This software is not stable, we consider it beta state and will
be in this stage until version 1.0.

Installation instructions
-------------------------

This program requires Python 3.4+.

Make a virtual environment:

``$ virtualenv allofplos``

Using pip:

``(allofplos)$ pip install allofplos``

This should install *allofplos* and requirements.

If you want to manually install from source (for example for development purposes), first clone the proyect repository:

``(allofplos)$ git clone git@github.com:PLOS/allofplos.git``

Install Python dependencies inside the newly created virtual environment:

``(allofplos)$ pip install -r requirements.txt``

How to run the program
----------------------

From inside the allofplos directory, run:

``(allofplos)$ python plos_corpus.py``

The first time it runs it will download a >4.4 Gb zip file
(**AllofPLOS\_article\_XML.zip**) with all the XML files inside.
**Note**: Make sure that you have enough space in your device for the
zip file and for it content before running this command (at least 30Gb).
After this file is downloaded, it will extract it contents into
AllofPLOS\_article\_XML directory.

If you ever downloaded the corpus before, it will make an incremental
update to the existing corpus, the script checks for and then downloads
to a temporary folder:

-  individual new articles that have been published
-  of those new articles, checks whether they are corrections (and
   whether the linked corrected article has been updated)
-  checks whether there are VORs (Versions of Record) for uncorrected
   proofs in the main articles directory & downloads those
-  checks whether the newly downloaded articles are uncorrected proofs
   or not after all of these checks, it moves the new articles into the
   main articles folder.

Here’s what the print statements might look like on a typical run:

::

    147 new articles to download.
    147 new articles downloaded.
    3 corrected articles found.
    0 corrected articles downloaded with new xml.
    Creating new text list of uncorrected proofs from scratch.
{'fraction_non_alphanumeric': 0.08338637810311904, 'fraction_numerical': 0.008062805007426267, 'mean_word_length': 3.7760891590678827, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5055793', 'n_tokens_mistral': 1254, 'n_tokens_neox': 1163, 'n_words': 659}