<commit_before>= How to submit data programatically
:doctype: book
:icons: font
:source-highlighter: highlightjs
:toc: left
:toclevels: 4
:sectlinks:

== Overview

This document will take you through the steps required to submit descriptive data (often called "metadata") using the
https://www.ebi.ac.uk[EMBL-EBI] submission system. EMBL-EBI provide a https://en.wikipedia.org/wiki/HATEOAS[hypermedia]
https://en.wikipedia.org/wiki/Representational_state_transfer[REST API] to accept descriptive data from submitters using
standard HTTP requests.

The examples in this walkthrough use https://en.wikipedia.org/wiki/CURL[curl] to interact with the server, but you can
use the API with any programming language.

The submission process follows these steps:

 1. Create a submission
 2. Add documents describing your data to the submission (samples, studies, etc...)
 3. Get feedback and amend documents if necessary
 4. Tell us the submission is ready to go

At this point we will send the data to the appropriate archive or archives. They will send reports on their progress
with your data and accessions back to us.

This guide assumes some familiarity with REST APIs and the HTTP protocol.

== Log in

*TODO* we will add a section describing how to get an authorisation token and what to do with it.

== Fetch the API root

We recommend that you start from the root endpoint and follow the _links_ given in the `_links` section of the
response, instead of hard-coding each API endpoint. This will insulate you from future changes to the submission system
architecture or URL structure.

include::{snippets}/root-endpoint/curl-request.adoc[]

This will return a long list of links, in http://stateless.co/hal_specification.html[HAL JSON] format:

include::{snippets}/root-endpoint/http-response.adoc[]

You don't need to worry about most of those links for now, we'll learn about them as we go.

== Create a submission

The submission system allows you to build up your submission over time and send it to the archives when you are ready.
The first step is to create a submission.

=== Pick a team

Every submission is owned by a _team_. A team is a group of people, working together on the same data. How you use this
is up to you. You could create a team for just yourself, everyone in a lab, or one for all collaborators on a project.

At this stage of development the API doesn't restrict what team you use. A future update will add support for team
membership and administration. Until then, you can use any team name you like.

A team document is structured like this:

`{"name" : "my-team"}`

At present, you also need to provide a submitter object when creating a submission. In future versions this will be
taken from your log in information, but for now you need to provide a submitter document:

`{"email": "<EMAIL>"}`

=== POSTing a submission

Using the link labelled `submissions:create`, `POST` a document containing a team and a submitter to create
a submission:

include::{snippets}/create-submission/curl-request.adoc[]

If successful, this will return a JSON object with links to parts of the submission:

include::{snippets}/create-submission/http-response.adoc[]

The `location` header in the response has the URL for the created submission. You will need this when adding things to
the submission.

== Adding samples

We'll start by adding samples. It's a good template for adding anything to a submission - while the details in the
documents change, the process is the same across all types of document.

=== Naming samples (and everything else)

Samples, like all the documents that you add to a submission need to have an _alias_. This field is the name used for
the record within the submission system. Each alias should be unique within the records of that type owned by a team.
Using an alias for a second time in another submission will cause the record to be overwritten.

=== Requirements for sample documents

The description of a sample should normally include a taxonomic code to describe the species classification. This must
be an entry from the taxonomy database curated at https://www.ncbi.nlm.nih.gov/taxonomy/[NCBI]. EMBL-EBI provide a
http://www.ebi.ac.uk/ena/browse/taxonomy-service[service] to search for and look up taxonomic entries and this is used
when the submission system checks sample documents.

We will accept sample records for display in https://www.ebi.ac.uk/biosamples/[BioSamples] without a taxon ID, but
you will not be able to use the sample record when submitting data the experimental data archives without a taxon ID.
In addition, if you use a taxon ID, it must be one marked as 'submittable' in the taxonomy database.

=== POSTing a sample

Using the link from the submission labelled `samples:create` and `POST` a sample object:

include::{snippets}/create-sample/curl-request.adoc[]

Note that the submission field above contains the URL to the submission, this allows the records to be linked together.

This will return a response to show that the sample record has been created:

include::{snippets}/create-sample/http-response.adoc[]

=== Validation

Descriptive data is checked at two points during submission:

 1. When sent to the API server, through `POST`, `PUT` and `PATCH` requests. At this point the sent must be match the
 expected syntax, and be identified with an alias.
 2. Once accepted by the API server, it is sent to a suite of validators to ensure it meets the requirements of the
 relevant archives or project.

The second stage of validation is more in depth, and takes longer. Clients can poll for `ValidationResults` for each
submitted document, and for a submission. You will not be able to complete a submission until validation
has completed and errors have been found.

Use the `validationResult` link from sample (or any other submitted document) to see its validation results. This may be
 pending, in which case you can poll for completion.

include::{snippets}/get-validation-result/curl-request.adoc[]

include::{snippets}/get-validation-result/http-response.adoc[]


=== Updating a sample

If you want to change a sample record after you've created it, you should use a `PUT` request to replace the record:

include::{snippets}/update-sample/curl-request.adoc[]

include::{snippets}/update-sample/http-response.adoc[]

Updating a document will cause it to be re-validated, as described above.

== Adding a study

Experimental data are grouped into _studies_. Studies should describe the motivation and summarise the design for the
study. Like samples, studies are identified by an alias, and have a number of attributes.

As with samples, study records are created using `POST` requests:

include::{snippets}/create-study/curl-request.adoc[]

This will return a response to show that the study record has been created:

include::{snippets}/create-study/http-response.adoc[]

== Adding assays

Assay documents describes the processes performed on the sample to produce data. For example, this could be library
preparation and sequencing for an NGS study. If you are familiar with the ENA SRA submission system, an assay is equivalent to an Experiment.

include::{snippets}/create-assay/curl-request.adoc[]

This will return a response to show that the assay record has been created:

include::{snippets}/create-assay/http-response.adoc[]

== Adding assay data

If you are familiar with the ENA SRA submission system, assay data is equivalent to a Run.

include::{snippets}/create-assay-data/curl-request.adoc[]

This will return a response to show that the assay record has been created:

include::{snippets}/create-assay-data/http-response.adoc[]

== Finalising your submission

Once you have added all your documents to the submission and they have been validated, you can finalise it and send them
to the archives. You do this by changing the submission status. To get to the submission status, follow the
`submissionStatus` link from the submission, then the `self` link.

Follow the `availableStatuses` link from submission or submissionStatus to see the statuses that are currently available.
Available statuses are affected by whever or not validation has finished and the current status of the submission.

Once validation is complete, you can change the submission status to 'Submitted', using either a `PUT` or `PATCH` request:

include::{snippets}/patch-submission-status/curl-request.adoc[]

This will return a response to show that the status has been changed:

include::{snippets}/patch-submission-status/http-response.adoc[]

This will also lock the submission, so you cannot make further changes while we broker the data into the archives.

== Tracking progress

Once you've finalised your submission, we will send it off to the relevant data archives. You can see how
that's progressing - e.g. has a study been received an accession yet?. There are several ways to do this.
<commit_msg>Added available status test case output to documentation<commit_after>= How to submit data programatically
:doctype: book
:icons: font
:source-highlighter: highlightjs
:toc: left
:toclevels: 4
:sectlinks:

== Overview

This document will take you through the steps required to submit descriptive data (often called "metadata") using the
https://www.ebi.ac.uk[EMBL-EBI] submission system. EMBL-EBI provide a https://en.wikipedia.org/wiki/HATEOAS[hypermedia]
https://en.wikipedia.org/wiki/Representational_state_transfer[REST API] to accept descriptive data from submitters using
standard HTTP requests.

The examples in this walkthrough use https://en.wikipedia.org/wiki/CURL[curl] to interact with the server, but you can
use the API with any programming language.

The submission process follows these steps:

 1. Create a submission
 2. Add documents describing your data to the submission (samples, studies, etc...)
 3. Get feedback and amend documents if necessary
 4. Tell us the submission is ready to go

At this point we will send the data to the appropriate archive or archives. They will send reports on their progress
with your data and accessions back to us.

This guide assumes some familiarity with REST APIs and the HTTP protocol.

== Log in

*TODO* we will add a section describing how to get an authorisation token and what to do with it.

== Fetch the API root

We recommend that you start from the root endpoint and follow the _links_ given in the `_links` section of the
response, instead of hard-coding each API endpoint. This will insulate you from future changes to the submission system
architecture or URL structure.

include::{snippets}/root-endpoint/curl-request.adoc[]

This will return a long list of links, in http://stateless.co/hal_specification.html[HAL JSON] format:

include::{snippets}/root-endpoint/http-response.adoc[]

You don't need to worry about most of those links for now, we'll learn about them as we go.

== Create a submission

The submission system allows you to build up your submission over time and send it to the archives when you are ready.
The first step is to create a submission.

=== Pick a team

Every submission is owned by a _team_. A team is a group of people, working together on the same data. How you use this
is up to you. You could create a team for just yourself, everyone in a lab, or one for all collaborators on a project.

At this stage of development the API doesn't restrict what team you use. A future update will add support for team
membership and administration. Until then, you can use any team name you like.

A team document is structured like this:

`{"name" : "my-team"}`

At present, you also need to provide a submitter object when creating a submission. In future versions this will be
taken from your log in information, but for now you need to provide a submitter document:

`{"email": "<EMAIL>"}`

=== POSTing a submission

Using the link labelled `submissions:create`, `POST` a document containing a team and a submitter to create
a submission:

include::{snippets}/create-submission/curl-request.adoc[]

If successful, this will return a JSON object with links to parts of the submission:

include::{snippets}/create-submission/http-response.adoc[]

The `location` header in the response has the URL for the created submission. You will need this when adding things to
the submission.

== Adding samples

We'll start by adding samples. It's a good template for adding anything to a submission - while the details in the
documents change, the process is the same across all types of document.

=== Naming samples (and everything else)

Samples, like all the documents that you add to a submission need to have an _alias_. This field is the name used for
the record within the submission system. Each alias should be unique within the records of that type owned by a team.
Using an alias for a second time in another submission will cause the record to be overwritten.

=== Requirements for sample documents

The description of a sample should normally include a taxonomic code to describe the species classification. This must
be an entry from the taxonomy database curated at https://www.ncbi.nlm.nih.gov/taxonomy/[NCBI]. EMBL-EBI provide a
http://www.ebi.ac.uk/ena/browse/taxonomy-service[service] to search for and look up taxonomic entries and this is used
when the submission system checks sample documents.

We will accept sample records for display in https://www.ebi.ac.uk/biosamples/[BioSamples] without a taxon ID, but
you will not be able to use the sample record when submitting data the experimental data archives without a taxon ID.
In addition, if you use a taxon ID, it must be one marked as 'submittable' in the taxonomy database.

=== POSTing a sample

Using the link from the submission labelled `samples:create` and `POST` a sample object:

include::{snippets}/create-sample/curl-request.adoc[]

Note that the submission field above contains the URL to the submission, this allows the records to be linked together.

This will return a response to show that the sample record has been created:

include::{snippets}/create-sample/http-response.adoc[]

=== Validation

Descriptive data is checked at two points during submission:

 1. When sent to the API server, through `POST`, `PUT` and `PATCH` requests. At this point the sent must be match the
 expected syntax, and be identified with an alias.
 2. Once accepted by the API server, it is sent to a suite of validators to ensure it meets the requirements of the
 relevant archives or project.

The second stage of validation is more in depth, and takes longer. Clients can poll for `ValidationResults` for each
submitted document, and for a submission. You will not be able to complete a submission until validation
has completed and errors have been found.

Use the `validationResult` link from sample (or any other submitted document) to see its validation results. This may be
 pending, in which case you can poll for completion.

include::{snippets}/get-validation-result/curl-request.adoc[]

include::{snippets}/get-validation-result/http-response.adoc[]


=== Updating a sample

If you want to change a sample record after you've created it, you should use a `PUT` request to replace the record:

include::{snippets}/update-sample/curl-request.adoc[]

include::{snippets}/update-sample/http-response.adoc[]

Updating a document will cause it to be re-validated, as described above.

== Adding a study

Experimental data are grouped into _studies_. Studies should describe the motivation and summarise the design for the
study. Like samples, studies are identified by an alias, and have a number of attributes.

As with samples, study records are created using `POST` requests:

include::{snippets}/create-study/curl-request.adoc[]

This will return a response to show that the study record has been created:

include::{snippets}/create-study/http-response.adoc[]

== Adding assays

Assay documents describes the processes performed on the sample to produce data. For example, this could be library
preparation and sequencing for an NGS study. If you are familiar with the ENA SRA submission system, an assay is equivalent to an Experiment.

include::{snippets}/create-assay/curl-request.adoc[]

This will return a response to show that the assay record has been created:

include::{snippets}/create-assay/http-response.adoc[]

== Adding assay data

If you are familiar with the ENA SRA submission system, assay data is equivalent to a Run.

include::{snippets}/create-assay-data/curl-request.adoc[]

This will return a response to show that the assay record has been created:

include::{snippets}/create-assay-data/http-response.adoc[]

== Finalising your submission

Once you have added all your documents to the submission and they have been validated, you can finalise it and send them
to the archives. You do this by changing the submission status. To get to the submission status, follow the
`submissionStatus` link from the submission, then the `self` link.

Follow the `availableStatuses` link from submission or submissionStatus to see the statuses that are currently available.
Available statuses are affected by whether or not validation has finished and the current status of the submission.

include::{snippets}/available-status-report/curl-request.adoc[]

This will return a response to show that the status can now be updated to 'Submitted':

include::{snippets}/available-status-report/http-response.adoc[]

Once validation is complete, you can change the submission status to 'Submitted', using either a `PUT` or `PATCH` request:

include::{snippets}/patch-submission-status/curl-request.adoc[]

This will return a response to show that the status has been changed:

include::{snippets}/patch-submission-status/http-response.adoc[]

This will also lock the submission, so you cannot make further changes while we broker the data into the archives.

== Tracking progress

Once you've finalised your submission, we will send it off to the relevant data archives. You can see how
that's progressing - e.g. has a study been received an accession yet?. There are several ways to do this.

{'fraction_non_alphanumeric': 0.06314682605567547, 'fraction_numerical': 0.0007748076816647296, 'mean_word_length': 4.582329317269076, 'pattern_counts': {'":': 2, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 12, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2014128', 'n_tokens_mistral': 4739, 'n_tokens_neox': 4486, 'n_words': 2547}