<commit_before># Wellcome Trust OA Compliance

## Data Model

### Overview

There are two components to the data model:

1. The spreadsheet which has been uploaded, which is an aggregator object for
2. The records that appear in the uploaded spreadsheet

For each (1) there can be many (2), and each (2) may be associated with exactly one (1)

### Spreadsheet Upload

```json

    {
        "id" : "<opaque identifier for upload>",
        "created_date" : "<date of upload of spreadsheet>",
        "filename" : "<original filename, as provided during upload>",
        "contact" : {
            "email" : "<contact email address>"
        },
        "status" : {
            "code" : "<current status of the processing job>",
            "message" : "<message for the user associated with the status>"
        }
    }

```

* id - will be used as part of the opaque url for the page from which downloads can be retrieved.
* contact.email - we use a nested object here in case we later want to store additional information about the contact
* status.code - the current status of the processing being done on the spreadsheet.  Could be:
    * submitted - spreadsheet has been uploaded but processing has not yet begun
    * processing - spreadsheet processing has begun
    * complete - spreadsheet processing has completed
    * error - there was a problem reading the spreadsheet
* status.message - human readable message for the user on the state of this upload.  Particularly useful if there is an error.

### Record

```json

    {
        "id" : "<opaque id of this record>",
        "created_date" : "<date this record was created">,
        "last_updated" : "<date this record was last modified>",
        
        "upload" : {
            "id" : "<opaque id of spreadsheet upload>",
            "pos" : <position of this record in the spreadsheet>
        },
        
        "source" : {
            "university" : "<university>",
            "pmcid" : "<pmcid>",
            "pmid" : "<pmid>",
            "doi" : "<doi>",
            "publisher" : "<publisher>",
            "journal_title" : "<journal title>",
            "article_title" : "<article title>",
            "apc" : "<total cost of apc>",
            "wellcome_apc" : "<amount of apc charged to wellcome oa grant>",
            "vat" : "<vat charged>",
            "total_cost" : "<total cost>",
            "grant_code" : "<wellcome grant code>",
            "licence_info" : "<original licence information>",
            "notes" : "<original notes>"
        },
        
        "identifiers" : {
            "pmcid" : "<canonical form of pmcid>",
            "pmid" : "<canonical form of pmid>",
            "doi" : <canonical form of doi>",
            "title" : "<article title>"
        },
        
        "supporting_info" : {
            "epmc_ft_xml" : true|false,
            "aam_from_ft_xml" : true|false,
            "aam_from_empc" : true|false,
            "issn" : ["<issn for this journal>"],
            "currently_in_oag" : true|false,
            "oag_pmcid" : "not_sent|sent|success|fto|error",
            "oag_doi" : "not_sent|sent|success|fto|error",
            "oag_pmid" : "not_sent|sent|success|fto|error",
        },
        
        "compliance" : {
            "in_epmc" : true|false,
            "epmc_is_oa" : true|false,
            "epmc_aam" : true|false,
            "licence" : {
                "type" : "<license type>"
            },
            "licence_source" : "epmc_xml|epmc|publisher",
            "journal_type" : "oa|hybrid",
            "confidence" : <out of 1>,
            "standard" : true|false,
            "deluxe" : true|false
        },
        
        "provenance" : [<commit_msg>fix typos in readme<commit_after># Wellcome Trust OA Compliance

## Data Model

### Overview

There are two components to the data model:

1. The spreadsheet which has been uploaded, which is an aggregator object for
2. The records that appear in the uploaded spreadsheet

For each (1) there can be many (2), and each (2) may be associated with exactly one (1)

### Spreadsheet Upload

```json

    {
        "id" : "<opaque identifier for upload>",
        "created_date" : "<date of upload of spreadsheet>",
        "filename" : "<original filename, as provided during upload>",
        "contact" : {
            "email" : "<contact email address>"
        },
        "status" : {
            "code" : "<current status of the processing job>",
            "message" : "<message for the user associated with the status>"
        }
    }

```

* id - will be used as part of the opaque url for the page from which downloads can be retrieved.
* contact.email - we use a nested object here in case we later want to store additional information about the contact
* status.code - the current status of the processing being done on the spreadsheet.  Could be:
    * submitted - spreadsheet has been uploaded but processing has not yet begun
    * processing - spreadsheet processing has begun
    * complete - spreadsheet processing has completed
    * error - there was a problem reading the spreadsheet
* status.message - human readable message for the user on the state of this upload.  Particularly useful if there is an error.

### Record

```json

    {
        "id" : "<opaque id of this record>",
        "created_date" : "<date this record was created>",
        "last_updated" : "<date this record was last modified>",
        
        "upload" : {
            "id" : "<opaque id of spreadsheet upload>",
            "pos" : "<integer: position of this record in the spreadsheet>"
        },
        
        "source" : {
            "university" : "<university>",
            "pmcid" : "<pmcid>",
            "pmid" : "<pmid>",
            "doi" : "<doi>",
            "publisher" : "<publisher>",
            "journal_title" : "<journal title>",
            "article_title" : "<article title>",
            "apc" : "<total cost of apc>",
            "wellcome_apc" : "<amount of apc charged to wellcome oa grant>",
            "vat" : "<vat charged>",
            "total_cost" : "<total cost>",
            "grant_code" : "<wellcome grant code>",
            "licence_info" : "<original licence information>",
            "notes" : "<original notes>"
        },
        
        "identifiers" : {
            "pmcid" : "<canonical form of pmcid>",
            "pmid" : "<canonical form of pmid>",
            "doi" : "<canonical form of doi>",
            "title" : "<article title>"
        },
        
        "supporting_info" : {
            "epmc_ft_xml" : true|false,
            "aam_from_ft_xml" : true|false,
            "aam_from_empc" : true|false,
            "issn" : ["<issn for this journal>"],
            "currently_in_oag" : true|false,
            "oag_pmcid" : "not_sent|sent|success|fto|error",
            "oag_doi" : "not_sent|sent|success|fto|error",
            "oag_pmid" : "not_sent|sent|success|fto|error",
        },
        
        "compliance" : {
            "in_epmc" : true|false,
            "epmc_is_oa" : true|false,
            "epmc_aam" : true|false,
            "licence" : {
                "type" : "<license type>"
            },
            "licence_source" : "epmc_xml|epmc|publisher",
            "journal_type" : "oa|hybrid",
            "confidence" : <out of 1>,
            "standard" : true|false,
            "deluxe" : true|false
        },
        
        "provenance" : [
{'fraction_non_alphanumeric': 0.12181379125301851, 'fraction_numerical': 0.0018781862087469815, 'mean_word_length': 2.0194410692588094, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4767313', 'n_tokens_mistral': 2147, 'n_tokens_neox': 2061, 'n_words': 698}