



  

COMMAND>\_ documentation

[![Logo](_static/command_logo.gif)](index.html#document-index)

 latest

 

Table of Contents

* [What is COMMAND>\_?](index.html#document-what_is_command)

* [Getting start with COMMAND>\_](index.html#document-start)

	+ [Getting my user id and password](index.html#getting-my-user-id-and-password)

	+ [Set up and select a compendium](index.html#set-up-and-select-a-compendium)

	+ [Searching public databases](index.html#searching-public-databases)

* [Message log](index.html#document-message_log)

* [Admin](index.html#document-admin_user)

	+ [Users/Group manager](index.html#users-group-manager)

	+ [Compendium manager](index.html#compendium-manager)

* [Deploy](index.html#document-deploy)

	+ [Requirements](index.html#requirements)

	+ [Docker Compose](index.html#id1)

	+ [Manual Deploy](index.html#manual-deploy)

* [Database schema](index.html#document-database_schema)

* [Use Cases](index.html#document-use_cases)

	+ [Use Case - Affymetrix from GEO](index.html#use-case-affymetrix-from-geo)

	+ [Use Case - Nimblegen from ArrayExpress](index.html#use-case-nimblegen-from-arrayexpress)

	+ [Use Case - Multiplatform Experiment](index.html#use-case-multiplatform-experiment)

	+ [Use Case - Import experiment from local file](index.html#use-case-import-experiment-from-local-file)

	+ [Use Case - RNA-Seq](index.html#use-case-rna-seq)

* [Mapping probes and export the gene expression matrix](index.html#mapping-probes-and-export-the-gene-expression-matrix)

* [Python parsing scripts](index.html#document-modules)

	+ [EXPERIMENT\_OBJECT variables](index.html#experiment-object-variables)

	+ [PLATFORM\_OBJECT variables](index.html#platform-object-variables)

	+ [SAMPLE\_OBJECT variables and methods](index.html#sample-object-variables-and-methods)

	+ [parsing\_scripts package](index.html#document-parsing_scripts)

* [COMMAND>\_ for developers](index.html#document-for_developers)

	+ [Add brand new feature in COMMAND>\_](index.html#add-brand-new-feature-in-command)

	+ [Add new public database manager](index.html#add-new-public-database-manager)

	+ [Add new compendium type](index.html#add-new-compendium-type)

	+ [Add new biological feature file importer](index.html#add-new-biological-feature-file-importer)

	+ [Add new platform type](index.html#add-new-platform-type)

	+ [Add new platform mapper](index.html#add-new-platform-mapper)

[COMMAND>\_](index.html#document-index)

* [Docs](index.html#document-index) »

* COMMAND>\_ documentation

* [Edit on GitHub](https://github.com/marcomoretto/command/blob/master/command/docs/index.rst)

---

Welcome to COMMAND>\_’s documentation![¶](#welcome-to-command-s-documentation "Permalink to this headline")

===========================================================================================================

COMMAND>\_ is a web-based application used to download, collect and manage gene expression data from public databases.

Main features are.

* Easy installation and update using [Docker Compose](https://docs.docker.com/compose/install/) technology.

* Graphical User Interface (GUI) for parsing and importing gene expression data.

* Default [Python](https://www.python.org/) scripts for easy parsing/importing of the most common microarray platforms (Affymetrix, Nimblegen, two-colors, etc.) and dedicated scripting editor for allowing flexible importing of any kind of gene expression data.

* Automatic pre-processing (downloading, trimming, mapping and counting) of bulk RNA-Seq data.

* Exporting of the collected data.

Note

Give it a try on <https://command.fmach.it:4242> using:

* username: `guest`

* password: `demo`

Check out the [Use Cases](index.html#document-use_cases)!

What is COMMAND>\_?[¶](#what-is-command "Permalink to this headline")

---------------------------------------------------------------------

COMMAND>\_[#f1]\_ is an acronym that stands for **COM**pendia **MAN**agement **D**esktop. It is the software used for the creation of several gene expression compendia such as COLOMBOS [[2]](#f2) and VESPUCCI [[3]](#f3). Despite being used since 2010 it has been made publicly available for anyone to use only in 2018, after having been completely rewritten.

COMMAND>\_ was originally conceived for the collection (and integration) of prokaryotes microarray experiments. As time goes by it has been evolved to allow also RNA-seq experiment to be imported and other species to be managed.

With the current implementation COMMAND>\_ is still meant for gene expression data collection but can be easily extended to support other kind of quantitative measurement technology (have a look at [COMMAND>\_ for developers](index.html#document-for_developers)).

COMMAND>\_ is a Python web application developed using the [Django](https://www.djangoproject.com/) framework for the backend, while the web interface has been developed using [ExtJS](https://www.sencha.com/products/extjs/#overview) with a look and feel typical of desktop applications. With COMMAND>\_ you can search and download experiment from public gene expression databases, such as [GEO](https://www.ncbi.nlm.nih.gov/gds) , [ArrayExpress](https://www.ebi.ac.uk/arrayexpress/) or [SRA](https://www.ncbi.nlm.nih.gov/sra), parse downloaded files to extract only valuable information, preview parsed data and import experiment data into a database. The pivotal point is the usage of custom Python scripts to mine only the relevant information. Scripts can be created or modified directly within the interface and are responsible to parse input files and populate each part of the **data model** (see [Database schema](index.html#document-database_schema)), i.e. measurement data and meta-data for *experiment*, *platforms* and *samples*.

For microarray platforms it would be necessary to map probes to genes but before this step genes have first to be imported. COMMAND>\_ allow to perform both these steps. For the latter it would be simply a matter of uploading a FASTA file with gene sequences (see data\_collection), while for the former a [BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi) alignment followed by a two-step filtering will be performed. In this way the microarray gets annotated with the latest available information enhancing the homogeneity since all microarrays will be annotated using the same gene list (see also map\_feature).

References

| [1] | Moretto, M. et al. (2019). First step toward gene expression data integration: transcriptomic data acquisition with COMMAND>\_. *BMC bioinformatics*, 20(1), 54. |

| [[2]](#id1) | Moretto, M. et al. (2015). COLOMBOS v3. 0: leveraging gene expression compendia for cross-species analyses. *Nucleic acids research*, 44(D1), D620-D623. |

| [[3]](#id2) | Moretto, M. et al. (2016). VESPUCCI: exploring patterns of gene expression in grapevine. *Frontiers in plant science*, 7, 633. |

Getting start with COMMAND>\_[¶](#getting-start-with-command "Permalink to this headline")

------------------------------------------------------------------------------------------

### Getting my user id and password[¶](#getting-my-user-id-and-password "Permalink to this headline")

If you are using the public COMMAND>\_ instance on <https://command.fmach.it:4242> you can login using:

> 

> * username: `guest`

> * password: `demo`

> 

> 

> 

This is a user with restricted privileges meant for demonstration purpose only. If you have your running instance of COMMAND>\_ (see [Deploy](index.html#document-deploy)) you will be able to first login using:

> 

> * username: `admin`

> * password: `admin`

> 

> 

> 

Now you can change the admin password, create new users and assign them privileges following the instructions in [Admin](index.html#document-admin_user).

![_images/login.png](_images/login.png)

### Set up and select a compendium[¶](#set-up-and-select-a-compendium "Permalink to this headline")

The first thing to do is creating a new empty compendium. Go to Admin (top bar) > Compendium Manager > Create Compendium (bottom-left corner + icon) and follow the instructions at [Admin](index.html#document-admin_user).

![_images/comp01.png](_images/comp01.png)

Now that a new compendium has been set up you need to retrieve a FASTA file containing the gene ids and sequences for the species you want to study.

Tip

For example you can visit the [NCBI Nucleotide database](https://www.ncbi.nlm.nih.gov/nucleotide/) and get the coding sequences for the organism of interest. This file is mandatory for blasting and mapping respectively in either microarray or RNA-Seq experiments. In order to import it into COMMAND>\_ go to Data collection (on the top-left corner) > Biological features, then select Import biological features from the bottom-left + icon.

![_images/comp02.png](_images/comp02.png)

Now your gene annotation file has been imported and you can start looking for interesting experiments (both microarray and RNA-Seq) related to the organism of interest.

### Searching public databases[¶](#searching-public-databases "Permalink to this headline")

After a new empty compendium has been created and a species of interest selected the user can start looking for collections of samples (from one or more experiments) from public databases (GEO, ArrayExpress or, in case of RNA-Seq experiments, SRA):

Go to > Data collection (on the top-left corner) then > Experiments > New Experiment (on the bottom right corner) > from public DB.

![_images/search01.png](_images/search01.png)

* In the Search options field of the dialog ‘Download from Public DB’ select the DB (here GEO) and the term of interest, either a description (e.g. Leukemia b-cell, Vitis vinifera, erc.) or directly a GSE ID.

* From the list select an experiment of interest and click the download button.

Tip

You can download multiple experiments at the same time.

![_images/search02.png](_images/search02.png)

* After a while, depending of the number of samples in the selected experiment(s) you have your experiment downloaded.

Tip

* Check Message log frequently.

* Inspect the Experiments section to see which experiments are available, yet to be parsed or already imported.

Now you can start parse and import some experiment (see [Use Cases](index.html#document-use_cases)).

Message log[¶](#message-log "Permalink to this headline")

---------------------------------------------------------

The Message Log page (Top > Options > Message Log) allows the user to take an eye on every activity of COMMAND>\_. Check it frequently!

[![admin](_images/message_log.png)](_images/message_log.png)

Admin[¶](#admin "Permalink to this headline")

---------------------------------------------

The admin interface is visible only to admin users that have complete access to COMMAND>\_ functionalities

and compendia.

### Users/Group manager[¶](#users-group-manager "Permalink to this headline")

[![admin](_images/admin_users.png)](_images/admin_users.png)

User manager page

The user menu allow to create, remove and modify users. Moreover, an admin user can assign users to groups

and set privileges to them. Group privileges are compendium-specific, i.e. we can for example restrict

access only to some compendia and avoid users belonging to a group to see the others. For those compendia

we can limit some functionalities, for example we could avoid users to run Python script or import experiments.

[![admin](_images/admin_privileges.png)](_images/admin_privileges.png)

Permission manager page

### Compendium manager[¶](#compendium-manager "Permalink to this headline")

[![admin](_images/admin_compendium.png)](_images/admin_compendium.png)

Compendium manager page

From this page a compendium can be created, modified, deleted and initialized. From a technical point of view

a *compendium* is nothing more than a database schema. When an admin user creates a new compendium he will be asked

to add all the information necessary to connect to such database.

[![admin](_images/admin_compendium_new.png)](_images/admin_compendium_new.png)

New compendium page

Once the connection information are saved and a new *compendium* appear in the grid, it would be possible to

*initialize* it, i.e. to create the database schema.

Note

In this way it is possible to have compendia hosted on different database server. If the database

do not exists yet it will be possible to have COMMAND>\_ to create it on the fly but you will be

asked to provide *username* and *password* for a **database admin user**.

Default database admin user is *postgres* with password *postgres*.

The *Compendium Type* section is read-only and at the moment is filled only with *gene expression* since

it is the only type of compendium you are allowed to create. To extend COMMAND>\_ and allow other kind

of quantitative data to be collected please have a look at [COMMAND>\_ for developers](index.html#document-for_developers).

Deploy[¶](#deploy "Permalink to this headline")

-----------------------------------------------

COMMAND>\_ is a complex application and relies on several other software components to work. In order to ease up the deployment process a `docker-compose.yml` file is provided, so assuming you have a working [Docker Compose](https://docs.docker.com/compose/) environment, the deployment process will be a matter of running a few commands.

In case you want to manually deploy COMMAND>\_ in your environment there will be more steps you will need to take care of such as installing the web-server, the DBMS, etc.

### Requirements[¶](#requirements "Permalink to this headline")

Have a look at the `requirements.txt` file for details. COMMAND>\_ main dependencies are:

> 

> * [Python 3](https://www.python.org/)

> * [Django](https://www.djangoproject.com/)

> * [PostgreSQL](https://www.postgresql.org/)

> * [Celery](http://www.celeryproject.org/)

> * [Channels](https://github.com/django/channels)

> * [Numpy](http://www.numpy.org/)

> * [Pandas](https://pandas.pydata.org/)

> * [BioPython](https://biopython.org/)

> 

> 

> 

COMMAND>\_ uses several `external tools` that you’ll need to download them separately:

> 

> * [AffxFusion.jar](https://github.com/HenrikBengtsson/Affx-Fusion-SDK/blob/master/affy/sdk/java/AffxFusion/dist/AffxFusion.jar)

> * [kallisto](https://pachterlab.github.io/kallisto/download)

> * [BLAST+](ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/)

> * [SRA-toolkit](https://www.ncbi.nlm.nih.gov/sra/docs/toolkitsoft/)

> * [Trimmomatic](http://www.usadellab.org/cms/?page=trimmomatic)

> 

> 

> 

### Docker Compose[¶](#id1 "Permalink to this headline")

Assuming that you have [Docker Compose correctly installed](https://docs.docker.com/compose/install/), you should be able to perform the following steps:

```

# 1. clone the repository

git clone https://github.com/marcomoretto/command.git

# 2. copy external dependencies (check figure below)

# 3. build

docker-compose build

# 4. start docker

docker-compose up -d

# 5. create database schema

docker-compose exec web python manage.py migrate

# 6. create admin user

docker-compose exec web python manage.py init_admin

# 7. create initial options

docker-compose exec web python manage.py init_options

# 8. create demo compendium

docker-compose exec web python manage.py init_demo_compendium demo

# 9. run daphne

docker-compose exec -d daphne daphne -b 0.0.0.0 -p 8001 cport.asgi:channel_layer

# 10. run worker

docker-compose exec -d worker python3 manage.py runworker

```

That’s it! You should be able to point your browser to <http://localhost> and login into COMMAND>\_ using:

* username: `admin`

* password: `admin`

Note

You should have the following directory structure for the `external tools`

![_images/external_dir_structure.png](_images/external_dir_structure.png)

Note

You might need to rename the directory from `command` to `cport` before doing step # 2.

### Manual Deploy[¶](#manual-deploy "Permalink to this headline")

One easy way to understand what you need to do to manually deploy COMMAND>\_ is to have a look at 2 files:

> 

> * the [Dockerfile](https://github.com/marcomoretto/command/blob/master/Dockerfile)

> * the [docker-compose.yml file](https://github.com/marcomoretto/command/blob/master/docker-compose.yml)

> 

> 

> 

In a nutshell, after having installed and configured [Nginx](https://www.nginx.com/) (or another web-server to run Django applications), [PostgreSQL](https://www.postgresql.org/), [Redis](https://redis.io/), [RabbitMQ](https://www.rabbitmq.com/) and [Celery](http://www.celeryproject.org/), you’ll have to run:

```

pip3 install --upgrade pip

pip3 install Cython==0.28.1

pip3 install -r requirements.txt

```

Now you should be ready configure Django (check the [documentation for details](https://docs.djangoproject.com/en/1.11)), create the database schema and run the application.

```

python manage.py migrate

python manage.py init_admin

python manage.py init_options

python manage.py init_demo_compendium demo

daphne -b 0.0.0.0 -p 8001 cport.asgi:channel_layer

python3 manage.py runworker

```

Note

COMMAND>\_ id a Django application so refer to the Django docs for database configuration <https://docs.djangoproject.com/en/1.11/ref/settings/>

Database schema[¶](#database-schema "Permalink to this headline")

-----------------------------------------------------------------

![_images/Database_schema.png](_images/Database_schema.png)

Use Cases[¶](#use-cases "Permalink to this headline")

-----------------------------------------------------

In this section we show how to both parse and import experiments from various gene expression platforms, technologies and sources (both public databases and local files) using the provided default scripts.

### Use Case - Affymetrix from GEO[¶](#use-case-affymetrix-from-geo "Permalink to this headline")

#### Import Gene Annotations[¶](#import-gene-annotations "Permalink to this headline")

We want to look for experiments related to Yeast: the [Saccharomyces Genome Database](https://www.yeastgenome.org/) is the proper choice for retrieving sequences associated to Yeast’s genes (from [this](https://downloads.yeastgenome.org/sequence/S288C_reference/orf_dna/orf_coding_all.fasta.gz) link).

Go to > Data collection (on the top left corner) then > Biological features > Import biological feature (+ symbol on the bottom left) > Type: FASTA , File name: select the annotation file you downloaded before > Import Biological features. Wait.

![_images/Case01_00.png](_images/Case01_00.png)

![_images/Case01_01.png](_images/Case01_01.png)

We start by selecting Experiments from Data collection (top left corner) then we highlight the experiment of interest (it was previously retrieved from GEO following [Searching public databases](index.html#search-database)), here [GSE8536](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE8536), an expression analyses study which inspects the response of Saccharomyces cerevisiae to stress throughout a 15-day wine fermentation.

#### Parse Experiment, Platform and Samples[¶](#parse-experiment-platform-and-samples "Permalink to this headline")

Since we have a new platform ([GPL90](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GPL90)) never imported before into COMMAND>\_ for this compendium, we retrieve the sequences associated to the Affymetrix probe ids ([`YG\_S98 probes`](_downloads/YG_S98.probe_tab)) for this platform from the [Affymetrix Support sitewebsite](http://www.affymetrix.com/support/technical/byproduct.affx?product=yeast).

From Experiments (Data collection Menu) we highlight the selected experiment (GSE8536 here) and click the Parse/Import experiment from the bottom bar.

On the main window you can see that the Experiment tab is populated with metadata gathered from the publicDB (GEO here).

Now we can start parsing the Experiment, the Platform(s) and the Samples.

![_images/Case01_02.png](_images/Case01_02.png)

![_images/Case01_03.png](_images/Case01_03.png)

Being a dataset retrieved from GEO we take advantage of the .soft file downloaded (see GEO Documentation for a description of this type of file):

Select GSE8536\_family.soft and click the Use assignment script to assign files to experiment entities icon on the bottom-right. A dialog will show-up:

* Script > assign\_all.py > Only selected files

* Experiment tab > Script: > soft\_experiment.py, Execution order: 1

* Platform tab > Script: > soft\_platform.py, Execution order: 1

* Sample tab > Script: > soft\_sample.py, Execution order: 1

* Run assignment script

![_images/Case01_03b.png](_images/Case01_03b.png)

Now in order to parse the new platform we are going to use the sequences associated to the Affymetrix probe ids we have already downloaded. We import the annotation ( [`YG\_S98 probes`](_downloads/YG_S98.probe_tab) ) in the File assignment section of Experiment files clicking the upload icon on the bottom of the page.

Now we associate the file to the platform:

* In Experiment files Section > File Assignment select the uploaded file (YG\_S98.probe\_tab) and click Use assignment script to assign files to experiment entities. On the Assign files dialog:

* Script: assign\_all.py

* Param:

* Only selected files checked (default)

* Platform tab > Script: gpr\_platform.py , Parameters: 0,Probe X|Probe Y,Probe Sequence , Execution order: 2

* Run assignment script

![_images/Case01_03c.png](_images/Case01_03c.png)

Note

* the Parameters assigned to the gpr\_platform.py script specify to not skip any line, use the combination of Probe X and Probe Y columns to create an unique id for the cel files and indicate the sequences for the probes are in the Probe Sequence column.

* The parsing of the Platform is a once time procedure: from now on we can use this platform for all related experiments.

Now we parse the Affymetrix cel files (sample files):

* In Experiment files Section > File Assignment we use CEL as filter and select all files > click the Use assignment script to assign files to experiment entities icon on the bottom-right corner and the Assign files and scripts to experiment structure dialog will pop-up:

* Script > match\_entity\_name.py

* Only selected files (default) checked

* Sample tab > Script: cell\_sample.py, Execution order: 2

* Run assignment script

![_images/Case01_03d.png](_images/Case01_03d.png)

Finally, in the Preview Section (Preview of GSE8536 here) click Run Selected (bottom-right corner). After a while your samples will be parsed.

Now you can Import both the Platform (since is the first time we use this specific one) and the Experiment.

Tip

Check that both the platform and the samples are properly parsed from the Preview interface of the Parse Experiment section clicking on the platform and on each sample.

Click the Import button on the bottom-right corner and select Import whole experiment. After a while the experiment and the platform (in this case) will be imported.

![_images/Case01_04.png](_images/Case01_04.png)

### Use Case - Nimblegen from ArrayExpress[¶](#use-case-nimblegen-from-arrayexpress "Permalink to this headline")

In COMMAND>\_ the preferred way to import experiments from public db is by using GEO which provide the most convenient interface out-of-the-box. In case an experiment is not included in GEO it is possible to import it from ArrayExpress. Start by searching the experiment of interest following the procedure described in [Searching public databases](index.html#search-database), select [E-GEOD-58806](https://www.ebi.ac.uk/arrayexpress/experiments/E-GEOD-58806/) as Term and ArrayExpress as Database. Go the experiment slide on the left, select the experiment of interest (here [E-GEOD-58806](https://www.ebi.ac.uk/arrayexpress/experiments/E-GEOD-58806/) ) and click >\_ Parse/Import experiment. On the main window you can see that the Experiment tab is populated with metadata gathered from the publicDB (ArrayExpress here).

#### Import Platform from GEO[¶](#import-platform-from-geo "Permalink to this headline")

COMMAND>\_can use a previous imported platform from a different public database (either ArrayExpress or GEO) and assign it as Reporter platform (in the preview main section of Parsing) for the current experiment.

In our case we want to parse and import an experiment from ArrayExpress using a previously imported platform from GEO.

In order to do so we import ONLY the platform for another experiment (here [GSE32561](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE32561)) which uses the same platform of the experiment of interest.

After the selection of the new experiment using the Searching from public db procedure we use the Nimblegen ndf files which allows to associate probes to sequences to the platform GPL14649.

Experiment files > File Assignment > Select GPL14649\_071112\_Ecoli\_K12\_EXP.ndf and in the Assign files dialog:

* Script: match\_entity\_name.py

* Param: platform

* Only selected files checked (default)

* Platform tab > Script: > gpr\_platform.py; Parameters: 0,X|Y,PROBE\_ID; Execution order: 2

* Run assignment script

![_images/Case02_01.png](_images/Case02_01.png)

Now we can import this platform from the Platform section of Preview:

![_images/Case02_02.png](_images/Case02_02.png)

#### Parse Experiment, Platform and Samples[¶](#id2 "Permalink to this headline")

Now the Platform is available and can be used to import the experiment retrieved from ArrayExpress.

Go to Experiments > Parse Experiment E-GEOD-58806 > Experiment Files > Platform and now click over A-GEOD-14649 in the Reporter Platform field and selected the previously imported GPL14649.

![_images/Case02_03.png](_images/Case02_03.png)

Finally you parse and import the nimblegen .pair files:

* In Experiment files Section > File Assignment > Filter .pair and select all files

* click the Use assignment script to assign files to experiment entities icon on the bottom-right and the Assign files and scripts to experiment structure dialog will pop-up:

* Script: match\_sample\_name.py > Only selected files

* Sample: Script: > pair\_sample.py, Execution order: 2

* Run assignment script

### Use Case - Multiplatform Experiment[¶](#use-case-multiplatform-experiment "Permalink to this headline")

It is standard practice for gene expressione esperiments to make use of multiple platforms for the same organism in the same experiment: usually it comes from multiple single experiments performed in different conditions/time. Here, we select from GEO the [GSE13713](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE13713) experiment regarding Phenotypic and transcriptomic analyses of mildly and severely salt-stressed Bacillus cereus ATCC. It is related to two platforms: [GPL7634](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GPL7634) and [GPL7636](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE7636).

#### Import Gene Annotation[¶](#import-gene-annotation "Permalink to this headline")

Since the platforms related to the selected experiment were never imported before into COMMAND>\_, we need the gene sequences in order to properly import our probes at gene level. We got gene/sequence list from ncbi: go [here](https://www.ncbi.nlm.nih.gov/nuccore/AP007209.1) and from the top-right button select send to: Coding sequences, Format: FASTA Nucleotide and Choose destination: File.

In COMMAND>\_ go to > Data Collection (on the top left corner) then > Bio features (genes) > Import biological feature (+ symbol on the bottom left) > Type: FASTA , File name: select the annotation file you downloaded before > Import Biological features.

![_images/Case03_01.png](_images/Case03_01.png)

#### Parse Platforms and Samples[¶](#parse-platforms-and-samples "Permalink to this headline")

In order to parse the two platforms, we need both the soft file related to the experiment and the soft\_platform.py script.

In Experiment files Section > File Assignement > Select the GSE13713\_family.soft file and on the Assign files dialog:

* Script: match\_all.py

* Param: platform

* Only selected files checked

* Platform tab > Script: > soft\_platform.py, parameters: True, Execution order: 1

In Experiment files Section > File Assignement > Select the .txt files (all Sultana in the Filter field) and on the Assign files dialog:

* Script: match\_entitye\_name.py

* Parameters: ch1

* Only selected files checked

Platform tab

* Script: gpr\_sample.py

* parameters: Gene name,Spot Mean Intensity (Cyanine5\_060909\_1136(1)),0

* Execution order: 2

Do the same again for the ch2 but use as Parameters for Platform:

Platform tab

* Script: gpr\_sample.py

* parameters: Gene name,Spot Mean Intensity (Cyanine3\_060909\_1136(1)),0

* Execution order: 2

for Platform GPL10439:

* In Experiment files Section > File Assignement > Select the .ndf file and on the Assign files dialog”:

> 

> 

> 	+ Script: match\_entity\_type\_param.py

> 	+ Param: platform

> 	+ Only selected files checked

> 	+ Platform tab > Script: > soft\_platform.py, Execution order: 2

>

* In Experiment files Section > File Assignement > Select the .txt files (all pair files) and on the Assign files dialog:

> 

> 

> 	+ Script: match\_entity\_name.py

>

* Parameters: ch1

> 

> 

> 	+ Only selected files checked

> 	+ Platform tab > Script: > gpr\_sample.py; Execution; order: 2

> 	+ Parameters: ID\_REF,Spot Mean Intensity (Alexa555\_101810\_0935(1)),0

>

* Parameters: ch2

> 

> 

> 	+ Only selected files checked

> 	+ Platform tab > Script: > gpr\_sample.py; Execution; order: 2

> 	+ Parameters: ID\_REF,Spot Mean Intensity (Alexa647\_111510\_1227(1))

>

### Use Case - Import experiment from local file[¶](#use-case-import-experiment-from-local-file "Permalink to this headline")

In order to import an experiment which is not available from public repositories the user needs to provide:

* a yaml file (see an example: [`here`](_downloads/experiment_example.yaml)) containing the description of the experiment to be imported:

The first row contains the Experiment id, the other rows start with the Platform id followed by the Samples ids.

* a single compressed file (either zip or tar.gz) containing the raw data.

Go to Experiments > New Experiment (bottom-left) > From local file

![_images/local_01.png](_images/local_01.png)

Fill the form which popped up starting with Experiment ID (the same contained in the yaml file, GSE13713 for the embedded example) then upload the yaml file (the system will take care to check if the format is ok), finally upload the compressed data. In a while your experiment is going to be imported.

![_images/local_02.png](_images/local_02.png)

![_images/local_03.png](_images/local_03.png)

![_images/local_04.png](_images/local_04.png)

### Use Case - RNA-Seq[¶](#use-case-rna-seq "Permalink to this headline")

Similarly to the microarray cases, RNA-Seq experiments can be retrieved from public database, specifically the [Sequence Read Archive (SRA)](https://www.ncbi.nlm.nih.gov/sra) , from the New Experiment/From public DB interface (bottom-left border icon).

Here we select a small RNA-Seq experiment from SRA ([PRJNA471071](https://www.ncbi.nlm.nih.gov/bioproject/PRJNA471071)) where the authors employed a computational model of underground metabolism and laboratory evolution experiments to examine the role of enzyme promiscuity in the acquisition and optimization of growth on predicted non-native substrates in E. coli K-12 MG1655.

![_images/CaseRNASeq_01.png](_images/CaseRNASeq_01.png)

#### Indexing[¶](#indexing "Permalink to this headline")

The first step is to build the index for the quasi-alignment mapper ([kallisto](https://pachterlab.github.io/kallisto/) here [[1]](#f1)):

select demo.fasta, It contains the sequences for the genes of the Escherichia coli genome and it is automatically build by COMMAND>\_ when you begin parsing the data.

Use Assignment Script (bottom-right corner icon) > from the dialog:match\_entity\_name.py > Only selected files

Experiment tab > Script: > kallisto\_index.py, Execution order: 1 > Run assignment script

#### RNA-Seq pre-processing and summarization[¶](#rna-seq-pre-processing-and-summarization "Permalink to this headline")

Since the experiment is paired-end, the default script for preprocessing and summarization requires to indicate only one of the two paired files.

You can do it using the filter and selecting \*1.fastq, the script will take care of the rest.

Use Assignment Script (bottom-right corner icon) > from the dialog:match\_entity\_name.py > Only selected files

Experiment tab > Script: > trim\_quantify.py, Execution order: 1, Parameters: 1 (being a paired end)

![_images/CaseRNASeq_05.png](_images/CaseRNASeq_05.png)

#### Run assignment script[¶](#run-assignment-script "Permalink to this headline")

After a while all the sample will be preprocessed and summarized and the experiment can be imported from the Preview section: bottom-right corner > Import whole experiment.

Mapping probes and export the gene expression matrix[¶](#mapping-probes-and-export-the-gene-expression-matrix "Permalink to this headline")

-------------------------------------------------------------------------------------------------------------------------------------------

If you are done with importing experiments you can now map the probes to genes using BLAST [[2]](#f2) and a double filtering GUI of COMMAND>\_.

Go to Platform, select the platform to be mapped (e.g. GPL90 from the Affymetrix Use Case) and click the chain icon (map platform to biological features) on the bottom left corner.

Now you can use the dialog to run BLAST and filter the data (here we use the default settings).

When your are fine with filtering you can use one of the selected filtered objects and download the expression matrix going to Options > Export.

![_images/MapProbes_01.png](_images/MapProbes_01.png)

![_images/MapProbes_02.png](_images/MapProbes_02.png)

Tip

You can filter the data with different parameters, each set of parameters is saved in a specific slot.

References

| [[1]](#id3) | Nicolas L Bray, Harold Pimentel, Páll Melsted and Lior Pachter, Near-optimal probabilistic RNA-seq quantification, Nature Biotechnology 34, 525–527 (2016), doi:10.1038/nbt.3519 |

| [[2]](#id4) | Altschul, S.F., Gish, W., Miller, W., Myers, E.W. & Lipman, D.J. (1990) “Basic local alignment search tool.” J. Mol. Biol. 215:403-410. |

Python parsing scripts[¶](#module-parsing_scripts.README "Permalink to this headline")

--------------------------------------------------------------------------------------

The Experiment Object, Platform Object and Sample Object are Python objects used as proxy to import a new experiment in the database

The file name of the experiment, platform or sample is stored in the variable named **INPUT\_FILE**

The name of the entity (experiment, platform or sample name) is stored in the variable named **ENTITY\_NAME**

To access parameters passed to each script use the list **PARAMETERS**

Within each entity (experiment, platform or sample) you can choose the execution order of the script using the Order column.

To access Experiment Object use the **EXPERIMENT\_OBJECT** variable in the Python script used with experiment files.

### EXPERIMENT\_OBJECT variables[¶](#experiment-object-variables "Permalink to this headline")

**EXPERIMENT\_OBJECT**.experiment\_access\_id: (string) the experiment access id

**EXPERIMENT\_OBJECT**.experiment\_name: (string) the experiment name

**EXPERIMENT\_OBJECT**.scientific\_paper\_ref: (string) pubblication associated to the experiment

**EXPERIMENT\_OBJECT**.description: (string) the experiment description

To access Platform Object use the **PLATFORM\_OBJECT** variable in the Python script used with platform files.

### PLATFORM\_OBJECT variables[¶](#platform-object-variables "Permalink to this headline")

**PLATFORM\_OBJECT**.platform\_access\_id (string) the platform access id

**PLATFORM\_OBJECT**.platform\_name (string) the platform name

**PLATFORM\_OBJECT**.platform\_type (string) ‘microarray or rna-seq’

**PLATFORM\_OBJECT**.description (string) the platform description

**PLATFORM\_OBJECT**.add\_bio\_feature\_reporter\_data(name, description, [\*\*](#id1)kwargs): add a reporter to the platform

[\*\*](#id3)kwargs are platform\_type dependent. i.e. for ‘microarray’ they are probe\_access\_id,

probe\_set\_name, probe\_type and sequence

To access Sample Object use the **SAMPLE\_OBJECT** variable in the Python script used with sample files.

### SAMPLE\_OBJECT variables and methods[¶](#sample-object-variables-and-methods "Permalink to this headline")

**SAMPLE\_OBJECT**.sample\_name (string) the sample name

**SAMPLE\_OBJECT**.description (string) the sample description

**SAMPLE\_OBJECT**.add\_raw\_data(bio\_feature\_reporter\_name, value): add raw data of this sample

### parsing\_scripts package[¶](#parsing-scripts-package "Permalink to this headline")

#### Subpackages[¶](#subpackages "Permalink to this headline")

##### parsing\_scripts.experiment package[¶](#parsing-scripts-experiment-package "Permalink to this headline")

###### Submodules[¶](#submodules "Permalink to this headline")

###### parsing\_scripts.experiment.kallisto\_index module[¶](#module-parsing_scripts.experiment.kallisto_index "Permalink to this headline")

`parsing_scripts.experiment.kallisto_index.``main`()[¶](#parsing_scripts.experiment.kallisto_index.main "Permalink to this definition")

Create an index file for the KALLISTO software using the current BIOLOGICAL FEATURES

Biologial features for this compendium are putted into a FASTA file that is then indexed

to be used for RNA-seq quantification using KALLISTO

PARAMETERS:

None

###### parsing\_scripts.experiment.soft\_experiment module[¶](#module-parsing_scripts.experiment.soft_experiment "Permalink to this headline")

`parsing_scripts.experiment.soft_experiment.``main`()[¶](#parsing_scripts.experiment.soft_experiment.main "Permalink to this definition")

Parse a SOFT file and extract EXPERIMENT information

Looks for **accession number**, **experiment name**, **scientific paper**, **experiment description**

PARAMETERS:

None

###### Module contents[¶](#module-parsing_scripts.experiment "Permalink to this headline")

##### parsing\_scripts.file\_assignment package[¶](#parsing-scripts-file-assignment-package "Permalink to this headline")

###### Submodules[¶](#submodules "Permalink to this headline")

###### parsing\_scripts.file\_assignment.assign\_all module[¶](#module-parsing_scripts.file_assignment.assign_all "Permalink to this headline")

`parsing_scripts.file_assignment.assign_all.``assign`(*input\_files*, *entity*, *entity\_type*, *parameters*)[¶](#parsing_scripts.file_assignment.assign_all.assign "Permalink to this definition")

Assign the selected input files (or all the files if checked) to every selected ENTITY

For each ENTITY (experiment, platforms or samples) for which a parsing script is selected, all the (selected)

input files will be assigned regardless.

PARAMETERS:

None

###### parsing\_scripts.file\_assignment.match\_entity\_name module[¶](#module-parsing_scripts.file_assignment.match_entity_name "Permalink to this headline")

`parsing_scripts.file_assignment.match_entity_name.``assign`(*input\_files*, *entity*, *entity\_type*, *parameters*)[¶](#parsing_scripts.file_assignment.match_entity_name.assign "Permalink to this definition")

Assign the selected input files (or all the files if checked) to every ENTITY with matching NAME

For each ENTITY (experiment, platforms or samples) for which a parsing script is selected, only the (selected)

input files with a name that match the one of the entity will be assinged

(for example a file name GSE123.soft would match the experiment entity GSE123).

PARAMETERS:

None

###### Module contents[¶](#module-parsing_scripts.file_assignment "Permalink to this headline")

##### parsing\_scripts.platform package[¶](#parsing-scripts-platform-package "Permalink to this headline")

###### Submodules[¶](#submodules "Permalink to this headline")

###### parsing\_scripts.platform.adf\_platform module[¶](#module-parsing_scripts.platform.adf_platform "Permalink to this headline")

`parsing_scripts.platform.adf_platform.``main`()[¶](#parsing_scripts.platform.adf_platform.main "Permalink to this definition")

Parse an ADF file and extract PLATFORM information

Looks for **accession number**, **platform name**, **platform type** and **platform description**

PARAMETERS:

*param1* (string): The original probe id field. If it is composed by more than one field, put all of them separated with a [|](#id1). For example X|Y

*param2* (bool): If True (or 1 or a non-empty string) the probe information (sequence) will be added

###### parsing\_scripts.platform.cdf\_platform module[¶](#module-parsing_scripts.platform.cdf_platform "Permalink to this headline")

`parsing_scripts.platform.cdf_platform.``main`()[¶](#parsing_scripts.platform.cdf_platform.main "Permalink to this definition")

Parse a CDF file (Affymetrix) and extract PLATFORM information

Looks for **probe set name** and **probe id**. Please note that CDF does not contain probe sequence, for that information refer to cdf\_platform\_fasta.py

PARAMETERS:

None

###### parsing\_scripts.platform.cdf\_platform\_fasta module[¶](#module-parsing_scripts.platform.cdf_platform_fasta "Permalink to this headline")

`parsing_scripts.platform.cdf_platform_fasta.``main`()[¶](#parsing_scripts.platform.cdf_platform_fasta.main "Permalink to this definition")

Parse a FASTA file containing probe sequences

This script is usually used before cdf\_platform.py in order to get the probe sequence information that a CDF file doesn’t provide.

PARAMETERS:

None

###### parsing\_scripts.platform.csv\_platform module[¶](#module-parsing_scripts.platform.csv_platform "Permalink to this headline")

`parsing_scripts.platform.csv_platform.``main`()[¶](#parsing_scripts.platform.csv_platform.main "Permalink to this definition")

Parse a CSV file containing probe sequences

A CSV file containing probe information is parsed and probes get added to the platform.

This script is usually used together with other PLATFORM scripts

PARAMETERS:

*param1* (string): The probe id field

*param2* (string): The probe sequence

###### parsing\_scripts.platform.gpr\_platform module[¶](#module-parsing_scripts.platform.gpr_platform "Permalink to this headline")

`parsing_scripts.platform.gpr_platform.``main`()[¶](#parsing_scripts.platform.gpr_platform.main "Permalink to this definition")

Parse a GPR file containing PLATFORM information and probe sequences

A GPR file is a TAB-delimited file with headers and complete platform information (descriptions and probe sequences)

PARAMETERS:

*param1* (int): Number of lines to skip

*param2* (string): The column header to parse out the original probe id field. If it is composed by more than one field, put all of them separated with a [|](#id3). For example X|Y (actual probe ids will be concatenated with dots . in that case)

*param3* (string): The column header of the probe **sequence** you want to parse out

*param4* (string): DEPRECATED - The column header to parse out the DB ‘gene\_map\_content’ field; if multiple seperate with a pipe | (actual probe ids will be concatenated with dots . in that case)

*param5* (string): The column header to parse out probe name field. If it is composed by more than one field, put all of them separated with a [|](#id5). For example X|Y (actual probe ids will be concatenated with dots . in that case)

*param6* (string): The column header to parse out probe set name field

*param7* (bool): Ensure that orgiginal probe id in SAMPLE\_OBJECT will be unique (defaults to False)

###### parsing\_scripts.platform.ndf\_platform module[¶](#module-parsing_scripts.platform.ndf_platform "Permalink to this headline")

`parsing_scripts.platform.ndf_platform.``main`()[¶](#parsing_scripts.platform.ndf_platform.main "Permalink to this definition")

Parse a NDF file containing probe sequences

A NDF file is an ArrayExpress file that contains probe sequences. They have a header file with X and Y position for the probe, the SEQUENCE field and a PROBE\_ID field.

The combination of X.Y is used to store the probe id and ensure that is a unique name

PARAMETERS:

*param1* (int): Number of lines to skip

###### parsing\_scripts.platform.soft\_platform module[¶](#module-parsing_scripts.platform.soft_platform "Permalink to this headline")

`parsing_scripts.platform.soft_platform.``main`()[¶](#parsing_scripts.platform.soft_platform.main "Permalink to this definition")

Parse a SOFT file and extract PLATFORM information

Looks for **accession number**, **platform name**, **platform type** and **platform description**. If True is passed as parameter it will look for

probe sequence information in the data table part of the file

PARAMETERS:

*param1* (bool): Read the data table information (default False)

###### Module contents[¶](#module-parsing_scripts.platform "Permalink to this headline")

##### parsing\_scripts.sample package[¶](#parsing-scripts-sample-package "Permalink to this headline")

###### Submodules[¶](#submodules "Permalink to this headline")

###### parsing\_scripts.sample.cel\_sample module[¶](#module-parsing_scripts.sample.cel_sample "Permalink to this headline")

`parsing_scripts.sample.cel_sample.``main`()[¶](#parsing_scripts.sample.cel_sample.main "Permalink to this definition")

Parse a CEL (Affymetrix) file and extract SAMPLE raw data

The probe original id is given by X.Y

PARAMETERS:

None

###### parsing\_scripts.sample.gpr\_sample module[¶](#module-parsing_scripts.sample.gpr_sample "Permalink to this headline")

`parsing_scripts.sample.gpr_sample.``main`()[¶](#parsing_scripts.sample.gpr_sample.main "Permalink to this definition")

Parse a GPR file and extract SAMPLE raw data

A GPR file is a TAB-delimited file with headers and complete sample raw data information

PARAMETERS:

*param1* (string): The column header of the original probe id to parse out. If it is composed by more than one field, put all of them separated with a [|](#id1). For example X|Y (actual probe ids will be concatenated with dots . in that case)

*param2* (string): The column header of the data value you want to parse out

*param3* (int): Number of lines to skip

*param4* (int): The sample channel (optional)

###### parsing\_scripts.sample.pair\_sample module[¶](#module-parsing_scripts.sample.pair_sample "Permalink to this headline")

`parsing_scripts.sample.pair_sample.``main`()[¶](#parsing_scripts.sample.pair_sample.main "Permalink to this definition")

Parse a PAIR file and extract SAMPLE raw data

A PAIR file is a TAB-delimited file with headers and complete sample raw data information

The probe id is given by X.Y to ensure uniqueness and the raw data value is taken from the PM column

PARAMETERS:

*param1* (int): Number of lines to skip

###### parsing\_scripts.sample.soft\_sample module[¶](#module-parsing_scripts.sample.soft_sample "Permalink to this headline")

`parsing_scripts.sample.soft_sample.``main`()[¶](#parsing_scripts.sample.soft_sample.main "Permalink to this definition")

Parse a SOFT file and extract SAMPLE description and optionally raw data

PARAMETERS:

*param1* (string): The raw data value field, if empty it will be assigned automatically using the sample\_column\_identifier function

###### parsing\_scripts.sample.trim\_quantify module[¶](#module-parsing_scripts.sample.trim_quantify "Permalink to this headline")

`parsing_scripts.sample.trim_quantify.``main`()[¶](#parsing_scripts.sample.trim_quantify.main "Permalink to this definition")

Trim a FASTQ file using Trimmomatic and quantify using KALLISTO

The result counts will be added to the SAMPLE OBJECT

PARAMETERS:

*param1* (bool): True if this FASTQ file has a PAIRED file (forward or reverse), default False

###### Module contents[¶](#module-parsing_scripts.sample "Permalink to this headline")

##### parsing\_scripts.utils package[¶](#parsing-scripts-utils-package "Permalink to this headline")

###### Submodules[¶](#submodules "Permalink to this headline")

###### parsing\_scripts.utils.column\_identifier module[¶](#module-parsing_scripts.utils.column_identifier "Permalink to this headline")

`parsing_scripts.utils.column_identifier.``sample_column_identifier`(*query*, *header*)[¶](#parsing_scripts.utils.column_identifier.sample_column_identifier "Permalink to this definition")

Tries to automatically identify the header column that contains the raw data given some query information (like the dye color)

Multi-channel array might have different dye color on different samples (dye-swap) and thus it would be tedious to manually

define it for each single sample. This function tries to do it for you and is tipically invoked for the SOFT sample files.

PARAMETERS:

*query* (string): The query string is usually something that contains information about the color i.e. cy3, red, green etc.

*header* (list): The header is a list of string from which to chose one that will match the query

###### parsing\_scripts.utils.rnaseq module[¶](#module-parsing_scripts.utils.rnaseq "Permalink to this headline")

`parsing_scripts.utils.rnaseq.``create_fasta`(*file*, *compendium*)[¶](#parsing_scripts.utils.rnaseq.create_fasta "Permalink to this definition")

Create a FASTA file using the BIOLOGICAL FEATURE of the current Organism

PARAMETERS:

*file* (string): The output FASTA file name

*compendium* (string): The organism (nick) name

###### Module contents[¶](#module-parsing_scripts.utils "Permalink to this headline")

#### Module contents[¶](#module-parsing_scripts "Permalink to this headline")

COMMAND>\_ for developers[¶](#command-for-developers "Permalink to this headline")

----------------------------------------------------------------------------------

In order to add new features to COMMAND>\_ you’ll need to stick with the whole framework. As a demonstration we will create a basic page to retrieve some data from the database and show them in a grid within COMMAND>\_.

So we will take care of:

> 

> * create the ExtJS interface;

> * create the Python view;

> * create the permission to access the view;

> * make an AJAX call passing parameters;

> * perform a job on celery to run in background;

> * handle websocket to show the results on a grid;

> 

> 

> 

We will also see how to extend COMMAND>\_ functionalities such as how to add a new public database users can use to perform search on, how to add a new platform type and so on.

Note

For anything else related to the interface design please refer to the [ExtJS documentation](https://docs.sencha.com/extjs/6.2.0/). While to properly add new models and extend the Data Model, please refer to the [Django documentation](https://docs.djangoproject.com/en/1.11/)

### Add brand new feature in COMMAND>\_[¶](#add-brand-new-feature-in-command "Permalink to this headline")

#### Create the ExtJS interface[¶](#create-the-extjs-interface "Permalink to this headline")

COMMAND>\_ is a single-page application, so everything you see runs within one [HTML file](https://github.com/marcomoretto/command/blob/master/templates/command/index.html) and the Javascript code needed to display the interface is loaded and managed by the ExtJS framework.

All ExtJS interface files (views) live within the directory `command/static/command/js/ext-js/app/view`. So let’s create a `test` directory in here and, within that directory let’s create 2 files: `Test.js` and `TestController.js`.

Let’s fill these two files with some basic code like the following:

```

// Test.js

Ext.define('command.view.test.Test', {

  extend: 'Ext.Component',

  xtype: 'test',

  title: 'Test',

  requires: [

      'Ext.panel.Panel',

      'command.view.test.TestController'

  ],

  controller: 'test',

  store: null,

  alias: 'widget.test',

  itemId: 'test',

  reference: 'test',

  viewModel: {},

  html: 'TEST',

  listeners: {

      //

  },

  initComponent: function() {

      this.callParent();

  },

  destroy: function() {

      this.callParent();

  }

});

```

```

// TestController.js

Ext.define('command.view.test.TestController', {

  extend: 'Ext.app.ViewController',

  alias: 'controller.test'

});

```

Now you will need to run the command `sencha app build` from within the `command/static/command/js/ext-js` directory.

Note

To use the `sencha app build` command you will need to download and install [Sencha CMD](https://docs.sencha.com/cmd/)

Now you should be able to point your browser to <http://localhost/#view/test> and see that the `Test` panel has been correctly loaded as a tab within the main application panel. To make it reachable with a button and to add a small icon next to the tab name we should edit two files, `Main.js` ([here](https://github.com/marcomoretto/command/blob/master/static/command/js/ext-js/app/view/main/Main.js)) and `Application.js` ([here](https://github.com/marcomoretto/command/blob/master/static/command/js/ext-js/app/Application.js)).

|  |  |

| --- | --- |

| 

```

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

```

 | 

```

// Main.js

// Add the ``Test`` menu button

},{

    text: 'Test',

    itemId: 'test\_menu\_item',

    iconCls: null,

    glyph: 'xf11b',

    listeners: {

        click: {

            fn: 'onAction',

            hash: 'view/test',

            glyph: 'xf11b',

            panel: 'test'

        }

    }

},{

    text: 'Options',

    ...

```

 |

|  |  |

| --- | --- |

| 

```

34

35

36

37

38

39

40

41

```

 | 

```

// Application.js

// Add the ``test`` glyph

version: null,

panel\_glyph: {

    'test': 'xf11b',

    ...

```

 |

You should see something like the following:

![_images/screenshot_test.png](_images/screenshot_test.png)

#### Create the Python View code[¶](#create-the-python-view-code "Permalink to this headline")

Now let’s create a grid, a basic double-click event and a link to a Python view. First of all we need to create the `test.py` file within the `views` directory ([here](https://github.com/marcomoretto/command/tree/master/command/lib/views)). The basic view file should look something like that:

```

 // test.py

 import json

 from django.http import HttpResponse

 from django.views import View

 from command.lib.utils.decorators import forward\_exception\_to\_http

class TestView(View):

   def get(self, request, operation, \*args, \*\*kwargs):

      method = getattr(self, operation)

      return method(request, \*args, \*\*kwargs)

   def post(self, request, operation, \*args, \*\*kwargs):

      method = getattr(self, operation)

      return method(request, \*args, \*\*kwargs)

   @staticmethod

   @forward\_exception\_to\_http

   def test(request, \*args, \*\*kwargs):

     return HttpResponse(json.dumps({'success': True}),

                         content\_type="application/json")

```

The `test` function does nothing at the moment and is meant to respond to an Ajax call. We’ll see that within the same `TestView` class we will put both code to manage Ajax and WebSocket requests. Before we add any business logic code we need to tell COMMAND>\_ that the ExtJS view `test` will make requests to the Python view `TestView` and that users need no specific privileges to do that (for the moment). So let’s add one line in the `consumer.py` script ([here](https://github.com/marcomoretto/command/blob/master/command/consumers.py)):

|  |  |

| --- | --- |

| 

```

34

35

36

37

38

39

40

41

```

 | 

```

# consumer.py

class Dispatcher:

   dispatcher = {

      ...

      ExportDataView: ['export\_data'],

      TestView: ['test']

   }

```

 |

#### Add a grid to the ExtJS interface[¶](#add-a-grid-to-the-extjs-interface "Permalink to this headline")

So far, so good. Let’s remove the HTML code from the `Test.js` file and let’s add a grid to show all the experiments for the selected compendium. The file will now look like this:

|  |  |

| --- | --- |

| 

```

 1

 2

 3

 4

 5

 6

 7

 8

 9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

```

 | 

```

// Test.js

Ext.define('command.view.test.Test', {

   extend: 'command.Grid',

   xtype: 'test',

   title: 'Test',

   requires: [

     'Ext.panel.Panel',

     'command.view.test.TestController'

   ],

   controller: 'test',

   store: null,

   alias: 'widget.test',

   itemId: 'test',

   reference: 'test',

   viewModel: {},

   mixins: {

      getRequestObject: 'RequestMixin'

   },

   command\_view: 'test',

   command\_read\_operation: 'test\_read',

   listeners: {

     //

   },

   columns: [{

     text: 'Accession',

     flex: 2,

     sortable: true,

     dataIndex: 'experiment\_access\_id',

   }, {

     text: 'Experiment name',

     flex: 2,

     sortable: true,

     tdCls: 'command\_tooltip',

     dataIndex: 'experiment\_name'

   }, {

     text: 'Scientific paper',

     flex: 2,

     sortable: true,

     dataIndex: 'scientific\_paper\_ref'

   }, {

     text: 'Description',

     flex: 2,

     sortable: true,

     tdCls: 'command\_tooltip',

     dataIndex: 'description'

   }],

   initComponent: function() {

     this.store = Ext.create('command.store.Experiments');

     this.callParent();

   },

   destroy: function() {

     this.callParent();

   }

});

```

 |

Please note that:

> 

> * at line `4` we extend `command.Grid`;

> * at line `31` we are saying to COMMAND>\_ the view to be used;

> * at line `33` we are declaring the default read operation (i.e. the default Python function to be called);

> * at line `64` we are declaring the ExtJS store to use.

> 

> 

> 

#### Link the ExtJS grid to the Python code via WebSocket[¶](#link-the-extjs-grid-to-the-python-code-via-websocket "Permalink to this headline")

The `test.py` Python view file will have a `test\_read` function that will look like the following:

```

# test.py

@staticmethod

@forward\_exception\_to\_channel

def test\_read(channel\_name, view, request, user):

   channel = Channel(channel\_name)

   start = 0

   end = None

   compendium = CompendiumDatabase.objects.get(id=request['compendium\_id'])

   if request['page\_size']:

      start = (request['page'] - 1) \* request['page\_size']

      end = start + request['page\_size']

   order = ''

   if request['ordering'] == 'DESC':

      order = '-'

   query\_response = Experiment.objects.using(compendium.compendium\_nick\_name). \

      filter(Q(experiment\_access\_id\_\_icontains=request['filter']) |

             Q(scientific\_paper\_ref\_\_icontains=request['filter']) |

             Q(description\_\_icontains=request['filter']) |

             Q(experiment\_name\_\_icontains=request['filter']))

   try:

      query\_response = query\_response.order\_by(order + request['ordering\_value'])

   except Exception as e:

      pass

   total = query\_response.count()

   query\_response = query\_response[start:end]

   channel.send({

      'text': json.dumps({

          'stream': view,

          'payload': {

              'request': request,

              'data': {

                  'experiments': [exp.to\_dict() for exp in query\_response],

                  'total': total

              }

          }

      })

   })

```

If you refresh your browser, you should now see something like the following:

![_images/screenshot_test_2.png](_images/screenshot_test_2.png)

As final step in this brief tutorial, let’s add a double-click event on the grid to call the `test` function defined in the `TestView` Python view to run an empty job on the Celery task manager. When the job is done we’ll have a callback function to show a message back on the interface. First thing is to add the event listener.

#### Create the Ajax call on double-click event[¶](#create-the-ajax-call-on-double-click-event "Permalink to this headline")

```

// Test.js

listeners: {

   itemdblclick: 'onTestDoubleClick'

},

```

Then we’ll need to implement the `onTestDoubleClick` in the `TestController.js`

```

// TestController.js

onTestDoubleClick: function(dv, record, item, index, e) {

     var grid = dv.up('grid');

     var gridSelection = grid.getSelection();

     var request = grid.getRequestObject('test');

     request.values = JSON.stringify(gridSelection[0].data);

     Ext.Ajax.request({

         url: request.view + '/' + request.operation,

         params: request,

         success: function (response) {

             command.current.checkHttpResponse(response);

         },

         failure: function (response) {

             console.log('Server error', reponse);

         }

     });

 }

```

#### Manage asynchronous code using Celery and WebSocket[¶](#manage-asynchronous-code-using-celery-and-websocket "Permalink to this headline")

The `request` object is configured to automatically retrieve the view name (`request.view`) and setted to call the `test` function in the Python `TestView`.

```

# test.py

 @staticmethod

 @forward\_exception\_to\_http

 def test(request, \*args, \*\*kwargs):

     values = json.loads(request.POST['values'])

     comp\_id = request.POST['compendium\_id']

     channel\_name = request.session['channel\_name']

     view = request.POST['view']

     operation = request.POST['operation']

     test.test\_task.apply\_async(

         (request.user.id, comp\_id, values['id'], channel\_name, view, operation)

     )

     return HttpResponse(json.dumps({'success': True}),

                         content\_type="application/json")

```

With the `test.test\_task.apply\_async` we are calling the `test\_task` function from the Celery task file `test.py` (not to be confused with the Python view file that have the same name). We need to create this file and implement the functionality. So let’s create a file name `test.py` in the `command/command/lib/task` directory ([here](https://github.com/marcomoretto/command/tree/master/command/lib/tasks)). The file will look like that:

```

# test.py

from \_\_future\_\_ import absolute\_import, unicode\_literals

from time import sleep

import celery

from channels import Channel

from command.lib.utils.message import Message

class TestCallbackTask(celery.Task):

    def on\_success(self, retval, task\_id, args, kwargs):

        user\_id, compendium\_id, path, channel\_name, view, operation = args

        channel = Channel(channel\_name)

        message = Message(type='info', title='Hello world!',

                          message='Hi there!'

                          )

        message.send\_to(channel)

    def on\_failure(self, exc, task\_id, args, kwargs, einfo):

        pass

@celery.task(base=TestCallbackTask, bind=True)

def test\_task(self, user\_id, compendium\_id, exp\_id, channel\_name, view, operation):

    sleep(1)

```

The `test\_task` function simply wait for one seconds. When it’s done the `on\_success` callback function gets called and it retrieve the WebSocket channel name to send back a simple message. That message will be captured on the client side and a pop-up will appear. Before trying it out we need to inform Celery that there’s an extra file to search for when calling a task. This is done in the Django setting file, [here](https://github.com/marcomoretto/command/blob/master/cport/settings.py).

```

# settings.py

CELERY\_IMPORTS = (

   'command.lib.tasks.experiment\_public',

   'command.lib.tasks.experiment\_local',

   'command.lib.tasks.uncompress\_file',

   'command.lib.tasks.run\_file\_assignment\_script',

   'command.lib.tasks.run\_parsing\_script',

   'command.lib.tasks.parse\_bio\_feature\_file',

   'command.lib.tasks.run\_platform\_mapper',

   'command.lib.tasks.import\_experiment',

   'command.lib.tasks.import\_platform\_mapping',

   'command.lib.tasks.export\_data',

   'command.lib.tasks.test',

)

```

You should now be able to double-click on a grid value and see something like this.

![_images/screenshot_test_3.png](_images/screenshot_test_3.png)

### Add new public database manager[¶](#add-new-public-database-manager "Permalink to this headline")

At the moment of writing, COMMAND>\_ is able to search on GEO, ArrayExpress and SRA.

![_images/screenshot_test_4.png](_images/screenshot_test_4.png)

To add a new database on this list, you will need to add a line in a database table and extend one class. In the `command\_datasource` database table you should add the source name and the class to handle it.

![_images/screenshot_test_5.png](_images/screenshot_test_5.png)

The class should be defined extending the class `PublicDatabase` that is defined [here](https://github.com/marcomoretto/command/blob/master/command/lib/coll/public_database.py). This is an abstract class and to extend it you will need to implement three methods:

> 

> * `search`: it perform the actual search on the public database (through a REST call or FTP for example) and create one `ExperimentSearchResult` for each retrieved entry to be stored in the database;

> * `download\_experiment\_files`: it is responsible to get all the data files related to one single `ExperimentSearchResult` and save them in the output directory;

> * `create\_experiment\_structure`: starting from the information of the downloaded files, this method should create the *experiment*, *platform*, *sample* structures and save it using `Experiment`, `Platform` and `Sample` Django models.

> 

> 

> 

### Add new compendium type[¶](#add-new-compendium-type "Permalink to this headline")

This is by far the easiest thing to do since it’s just a matter of adding one tuple on the `command` DB. The table to be modified is the `command\_compendiumtype` table. At the moment

the only compendium type defined is the gene expression one. The fields are *name*, *description* and the *biological feature name*, so respectively *gene\_expression*, *Gene expression compendium* and *gene*.

![_images/screenshot_db_compendium.png](_images/screenshot_db_compendium.png)

### Add new biological feature file importer[¶](#add-new-biological-feature-file-importer "Permalink to this headline")

All the classes releated to importing *biological features* are located [here](https://github.com/marcomoretto/command/tree/master/command/lib/coll/biological_feature). First thing to do is to

inform the dispatcher in the `importers.py` file which are the classes responsible to manage different file types. For example, genes will be imported using FASTA files. The second step is

to actually implement the class extending the `BaseImporter` class. The newly defined class will need to implement the `parse` method and redefine the `FILE\_TYPE\_NAME` variable.

```

# fasta\_file\_importer.py

class FastaFileImporter(BaseImporter):

     FILE\_TYPE\_NAME = 'FASTA'

     def parse(self, filename):

         sequence\_field = BioFeatureFields.objects.using(self.compendium).get(name='sequence')

         with transaction.atomic(using=self.compendium):

             with open(filename, 'rU') as handle:

                 for record in SeqIO.parse(handle, 'fasta'):

                     gene = BioFeature()

                     gene.name = record.id

                     gene.description = record.description

                     gene.save(using=self.compendium)

                     bf\_value = BioFeatureValues()

                     bf\_value.bio\_feature = gene

                     bf\_value.bio\_feature\_field = sequence\_field

                     bf\_value.value = str(record.seq)

                     bf\_value.save(using=self.compendium)

```

![_images/screenshot_fasta_importer.png](_images/screenshot_fasta_importer.png)

### Add new platform type[¶](#add-new-platform-type "Permalink to this headline")

To add a new platform type there are several step to do and mostly depends on the kind of platform is going to be added.

**Database entry**

To add a new platform type for a single compendium (organism) you will need to add a tuple with name, description, bio feature reporter name and the compendium type ID, for example:

*microarray*, *MicroArray*, *probe* and *1* to the `command\_platformtype` table. If you want *every* new compendium you are going to create to have such new platform you will need to add the same tuple to the

`command\_platformtypeadmin` table in the `command` DB.

![_images/screenshot_plt_type.png](_images/screenshot_plt_type.png)

**Reporters ExtJS GUI**

Next step will be to inform the GUI how to behave when the user wants to see the *biological feature reporters* associated with the new platform. For example in case of Microarray the

*biological feature reporters* are the probes. The file to modify is `PlatformController.js` (defined [here](https://github.com/marcomoretto/command/blob/master/static/command/js/ext-js/app/view/data_collection/platform/PlatformsController.js)).

`onViewBioFeatureReporter` is the method to modify adding a new case for the new platform. For example in case of RNA-seq we simply display a message to say there’s no associated

*biological feature reporters* since the gene expression measurement in this case is directly given by read counts. For Microarray instead we have probes and thus we will open a new

window to show the probes associated with this platform, the `window\_bio\_feature\_reporter` window.

```

// PlatformsController.js

onViewBioFeatureReporter: function (me) {

    var selection = me.up('grid').getSelectionModel().getSelection()[0].data;

    var comp = JSON.parse(localStorage.getItem("current\_compendium"));

    if (selection.platform\_type) {

        switch (selection.platform\_type.name) {

            case 'rnaseq':

                Ext.MessageBox.show({

                    title: 'RNA-seq platform',

                    msg: 'For RNA-seq platform ' + selection.platform\_access\_id +  ', ' +  comp.compendium\_type.bio\_feature\_name + ' is/are directly measured',

                    buttons: Ext.MessageBox.OK,

                    icon: Ext.MessageBox.INFO,

                    fn: function (a) {

                    }

                });

                break

            case 'microarray':

                var win = Ext.create({

                    xtype: 'window\_bio\_feature\_reporter',

                    title: 'Microarray platform ' + selection.platform\_access\_id + ': ' +

                        comp.compendium\_type.bio\_feature\_name + ' feature reporters (' + selection.platform\_type.bio\_feature\_reporter\_name + ')',

                    platform: selection

                });

                break

        }

    }

```

### Add new platform mapper[¶](#add-new-platform-mapper "Permalink to this headline")

When a platform has *biological feature reporters* associated, these must be mapped to the *biological features*. In case of *gene expression* compendium the *biological features* are genes.

So to give a concrete example we will need to associate Microarray probes to genes. This step is very platform-dependant and so a lot of freedom is left to the developer to design

the GUI. There are just few things to keep in mind in order to have everything working correctly within the COMMAND>\_ framework.

**Mapper ExtJS GUI**

First thing will be to inform the GUI how to behave when the user wants to map this platform reporters to the *biological features*. The file to modify is again the `PlatformController.js` (defined [here](https://github.com/marcomoretto/command/blob/master/static/command/js/ext-js/app/view/data_collection/platform/PlatformsController.js)),

but this time we are going to modify the `onMapPlatformToBioFeature` method, adding a new case for the new platform. For Microarray we defined a new window `window\_map\_microarray\_platform`

[here](https://github.com/marcomoretto/command/tree/master/static/command/js/ext-js/app/view/data_collection/platform/microarray). Again, in this case the developer is left completely free to design it as he wants.

```

// PlatformsController.js

onMapPlatformToBioFeature: function (me) {

    var selection = me.up('grid').getSelectionModel().getSelection()[0].data;

    var comp = JSON.parse(localStorage.getItem("current\_compendium"));

    if (selection.platform\_type) {

        switch (selection.platform\_type.name) {

            case 'rnaseq':

                Ext.MessageBox.show({

                    title: 'RNA-seq platform',

                    msg: 'RNA-seq platform ' + selection.platform\_access\_id +  ' is automatically mapped to ' + comp.compendium\_type.bio\_feature\_name,

                    buttons: Ext.MessageBox.OK,

                    icon: Ext.MessageBox.INFO,

                    fn: function (a) {

                    }

                });

                break

            case 'microarray':

                command.current.createWin({

                    xtype: 'window\_map\_microarray\_platform',

                    title: 'Map microarray platform ' + selection.platform\_access\_id + ' to ' + comp.compendium\_type.bio\_feature\_name,

                    platform: selection

                });

                break

        }

    }

```

**Mapper Django View**

The associated Django View is defined in `platform.py` view file [here](https://github.com/marcomoretto/command/blob/master/command/lib/views/platforms.py) and for Microarray this is the `MicroarrayPlatformView` class.

This is pretty standard view as described previously.

**Mapper code**

The actual code is stored in a class that will extend the `BaseMapper` (placeholder) class. For Microarray this class is `MicroarrayMapper` and is located [here](https://github.com/marcomoretto/command/blob/master/command/lib/coll/platform/microarray_mapper.py).

Last step is to inform the mapper dispatcher on which class to invoke, and this is done in the `mappers.py` file located [here](https://github.com/marcomoretto/command/blob/master/command/lib/coll/platform/mappers.py).

```

// mappers.py

from command.lib.coll.platform.microarray\_mapper import MicroarrayMapper

platform\_mapper = {

    'microarray': MicroarrayMapper

}

```

Contribute & Support[¶](#contribute-support "Permalink to this headline")

=========================================================================

Use the [GitHub Push Request and/or Issue Tracker](https://github.com/marcomoretto/command).

Author[¶](#author "Permalink to this headline")

===============================================

To send me an e-mail about anything else related to COMMAND>\_ write to ![email](_images/email.png)

License[¶](#license "Permalink to this headline")

=================================================

The project is licensed under the [GPLv3 license](https://www.gnu.org/licenses/gpl-3.0.html).

How to cite[¶](#how-to-cite "Permalink to this headline")

=========================================================

If you find COMMAND>\_ useful for your work please cite

Moretto, M., Sonego, P., Villaseñor-Altamirano, A. B., & Engelen, K. (2019). **First step toward gene expression data integration: transcriptomic data acquisition with COMMAND>\_.** *BMC bioinformatics*, 20(1), 54. ISO 690

<https://doi.org/10.1186/s12859-019-2643-6>

