<commit_before>:::: The Ensembl Functional Genomics Array Mapping Environment ::::

This document details the configuration and functionality available using the eFG 'arrays' 
environment, which utilises both the eFG pipeline environment and the Ensembl genebuild pipeline 
technology. The eFG environment provides configuration and command line access to various functions 
which can run the whole mapping pipeline or allow a more flexible step wise approach.

The eFG environment currently supports genomic and transcript alignment, and transcript annotation of
the following formats unless otherwise stated:

Format		  Description		Definition	   
AFFY_UTR	  Standard IVT		25mer Probesets anti-sense target
AFFY_ST		  Sense Target		25mer Probesets sense target
ILLUMINA_WG	  WholeGenome		50mer Probes sense target 
CODELINK_WG	  WholeGenome		30mer Probes sense target 
PHALANX		  OneArray			60mer Probes sense target
AGILENT		  Formats?			60mer Probes sense target

Not fully supported
LEIDEN	  <NAME>
NIMBLEGEN_TILING

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


Contents

1    Introduction
2    Overview
3    Pre-requisites
4    The Ensembl Pipeline
5    The eFG Arrays Environment
6    Input Data
7    Initiating An Instance
8    Running The Pipeline
8.1  Probe Alignment
8.2  Transcript Annotation
9    Administration Functions
10   Adding Additional Support
10.1 Adding A New Array Format
10.2 Adding A New Array
10.3 Tiling Arrays
10.4 Multi-Species Support
10.5 Unsuitable Species Name Support (SPECIES_COMMON) 
11   Known Issues/Caveats

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


1 Introduction

The array mapping pipeline consists of two distinct stages:

Probe Alignment

The probe sequences from a given array are aligned to the genomic sequence. If applicable (i.e. 
expression design) probes are also mapped to transcript sequences(cDNA).  These alignments are then 
mapped back to the genome and stored as gapped alignments, any ungapped alignments from this process 
are discarded as they will be represented by the genomic alignments. Transcript associations defined by 
this process are stored using a DBEntry object.

Transcript Annotation

Probe/sets are assigned to transcripts given a set of simple rules dependant on the array design.
Historically this has involved a 2KB extension of the 3' UTR sequence as it is known that the Ensembl 
gene build pipeline can be conservative in predicting UTRs. The new pipeline allows for much more 
flexible configuration taking into account species specific variation of UTRs. The current default 
strategy for annotating arrays is detailed here:

www.ensembl.org/info/docs/microarray_probe_set_mapping.html

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


2 Overview

If required, edit the following pipeline configurations files(see section 4)

ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/Config/General.pm
ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/Config/BatchQueue.pm.efg_arrays
ensembl-analysis/modules/Bio/EnsEMBL/Analysis/Config/ImportArrays.pm 
ensembl-analysis/modules/Bio/EnsEMBL/Analysis/Config/ProbeAlign.pm 

If required, edit the following environment files(see section 5):

ensembl-functgenomics/scripts/efg.config
ensembl-functgenomics/scripts/environments/pipeline.config
ensembl-functgenomics/scripts/environments/arrays.config

If required, set up input directory structure and data (see section 6) e.g.

$DATA_HOME/HOMO_SAPIENS/AFFY_UTR/HC-G110_probe_fasta     
$DATA_HOME/HOMO_SAPIENS/AFFY_UTR/HG-U133A_probe_fasta        
...

$DATA_HOME/HOMO_SAPIENS/AFFY_ST/HuEx-1_0-st-v2.probe.fa
$DATA_HOME/HOMO_SAPIENS/AFFY_ST/HuGene-1_0-st-v1.probe.fa

etc.

Create an instance file(see section 6), e.g.

my_human_54_37p.arrays

Initialise the environment and run the alignments and annotation (see section 7).
If required, run TestImportArrays and TestProbleAlign.
Check alignment reports before running transcript annotation.

>bash
>. ensembl-functgenomics/scripts/.efg
>. my_human_54_37p.arrays password
>RunAlignments
>RunTranscriptXrefs

Check transcript annotation logs to ensure probe2transcript has completed succesfully.
See section 8 for rollback and administration functions.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


3 Pre-requisites/Requirements

See pipeline.txt Pre-requisites.

exonerate-v2.2.0 can be found here :
http://www.ebi.ac.uk/~guy/exonerate/

Memory

This is dependant on several factors including the number of arrays to be mapped, the
format of the arrays and the size of the genome/transcriptome of a given species. For human, which 
tends to be the largest dataset Ensembl deals with, some steps can require upto 12GB of memory. However, 
this is handled dynamically by the environment dependant on the values of:

HUGEMEM_HOME	- Directory visible to huge memory host(this may be the same as DATA_HOME)
HUGEMEM_QUEUE	- LSF queue name for huge memory host
MAX_NORMAL_MEM  - Maximum memory usage on normal memory host
MAX_HUGE_MEM    - Maximum memory usage on huge memory host

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


4 The Ensembl Pipeline

See pipeline.txt for generic pipeline information and configuration.

The ensembl-analysis code deals with the actual work of a given analysis, with modules being split
into Runnables which perform the actual analysis and RunnableDBs which handle post processing and
interaction with the ensembl output DB e.g. your funcgen DB.  There are just three modules which 
perform the bulk of the array mapping analyses:

ensembl-analysis/modules/Bio/EnsEMBL/Analysis/RunnableDB/ImportArrays.pm - Runs as a single job to 
collapse arrays of the same format into a non-redundant set of probes, storing probe records in the 
output DB and writing a non-reundant probe fasta file to be used in the alignment step.

ensembl-analysis/modules/Bio/EnsEMBL/Analysis/Runnable/ExonerateProbe.pm - Performs exonerate 
alignments and generates and filters ProbeFeatures given a max 
mismatch value.

ensembl-analysis/modules/Bio/EnsEMBL/Analysis/RunnableDB/ProbeAlign.pm - Runs the ExonerateProbe 
Runnable using genomic or transcript target sequence, and performs post processing and storage of 
ProbeFeatures, UnmappedObjects and DBEntries. Dependant on the target sequence the analysis run by <commit_msg>Clarified editing of General.pm<commit_after>:::: The Ensembl Functional Genomics Array Mapping Environment ::::

This document details the configuration and functionality available using the eFG 'arrays' 
environment, which utilises both the eFG pipeline environment and the Ensembl genebuild pipeline 
technology. The eFG environment provides configuration and command line access to various functions 
which can run the whole mapping pipeline or allow a more flexible step wise approach.

The eFG environment currently supports genomic and transcript alignment, and transcript annotation of
the following formats unless otherwise stated:

Format		  Description		Definition	   
AFFY_UTR	  Standard IVT		25mer Probesets anti-sense target
AFFY_ST		  Sense Target		25mer Probesets sense target
ILLUMINA_WG	  WholeGenome		50mer Probes sense target 
CODELINK_WG	  WholeGenome		30mer Probes sense target 
PHALANX		  OneArray			60mer Probes sense target
AGILENT		  Formats?			60mer Probes sense target

Not fully supported
LEIDEN	  <NAME>
NIMBLEGEN_TILING

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


Contents

1    Introduction
2    Overview
3    Pre-requisites
4    The Ensembl Pipeline
5    The eFG Arrays Environment
6    Input Data
7    Initiating An Instance
8    Running The Pipeline
8.1  Probe Alignment
8.2  Transcript Annotation
9    Administration Functions
10   Adding Additional Support
10.1 Adding A New Array Format
10.2 Adding A New Array
10.3 Tiling Arrays
10.4 Multi-Species Support
10.5 Unsuitable Species Name Support (SPECIES_COMMON) 
11   Known Issues/Caveats

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


1 Introduction

The array mapping pipeline consists of two distinct stages:

Probe Alignment

The probe sequences from a given array are aligned to the genomic sequence. If applicable (i.e. 
expression design) probes are also mapped to transcript sequences(cDNA).  These alignments are then 
mapped back to the genome and stored as gapped alignments, any ungapped alignments from this process 
are discarded as they will be represented by the genomic alignments. Transcript associations defined by 
this process are stored using a DBEntry object.

Transcript Annotation

Probe/sets are assigned to transcripts given a set of simple rules dependant on the array design.
Historically this has involved a 2KB extension of the 3' UTR sequence as it is known that the Ensembl 
gene build pipeline can be conservative in predicting UTRs. The new pipeline allows for much more 
flexible configuration taking into account species specific variation of UTRs. The current default 
strategy for annotating arrays is detailed here:

www.ensembl.org/info/docs/microarray_probe_set_mapping.html

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


2 Overview

If required, edit the following pipeline configurations files(see section 4)

ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/Config/General.pm
ensembl-pipeline/modules/Bio/EnsEMBL/Pipeline/Config/BatchQueue.pm.efg_arrays
ensembl-analysis/modules/Bio/EnsEMBL/Analysis/Config/ImportArrays.pm 
ensembl-analysis/modules/Bio/EnsEMBL/Analysis/Config/ProbeAlign.pm 

If required, edit the following environment files(see section 5):

ensembl-functgenomics/scripts/efg.config
ensembl-functgenomics/scripts/environments/pipeline.config
ensembl-functgenomics/scripts/environments/arrays.config

If required, set up input directory structure and data (see section 6) e.g.

$DATA_HOME/HOMO_SAPIENS/AFFY_UTR/HC-G110_probe_fasta     
$DATA_HOME/HOMO_SAPIENS/AFFY_UTR/HG-U133A_probe_fasta        
...

$DATA_HOME/HOMO_SAPIENS/AFFY_ST/HuEx-1_0-st-v2.probe.fa
$DATA_HOME/HOMO_SAPIENS/AFFY_ST/HuGene-1_0-st-v1.probe.fa

etc.

Create an instance file(see section 6), e.g.

my_human_54_37p.arrays

Initialise the environment and run the alignments and annotation (see section 7).
If required, run TestImportArrays and TestProbleAlign.
Check alignment reports before running transcript annotation.

>bash
>. ensembl-functgenomics/scripts/.efg
>. my_human_54_37p.arrays password
>RunAlignments
>RunTranscriptXrefs

Check transcript annotation logs to ensure probe2transcript has completed succesfully.
See section 8 for rollback and administration functions.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


3 Pre-requisites/Requirements

See pipeline.txt Pre-requisites.

exonerate-v2.2.0 can be found here :
http://www.ebi.ac.uk/~guy/exonerate/

Memory

This is dependant on several factors including the number of arrays to be mapped, the
format of the arrays and the size of the genome/transcriptome of a given species. For human, which 
tends to be the largest dataset Ensembl deals with, some steps can require upto 12GB of memory. However, 
this is handled dynamically by the environment dependant on the values of:

HUGEMEM_HOME	- Directory visible to huge memory host(this may be the same as DATA_HOME)
HUGEMEM_QUEUE	- LSF queue name for huge memory host
MAX_NORMAL_MEM  - Maximum memory usage on normal memory host
MAX_HUGE_MEM    - Maximum memory usage on huge memory host

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


4 The Ensembl Pipeline

See pipeline.txt for generic pipeline information and configuration. Note: General.pm does not need 
editing unless you want to specifically modify some of the General pipeline config.

The ensembl-analysis code deals with the actual work of a given analysis, with modules being split
into Runnables which perform the actual analysis and RunnableDBs which handle post processing and
interaction with the ensembl output DB e.g. your funcgen DB.  There are just three modules which 
perform the bulk of the array mapping analyses:

ensembl-analysis/modules/Bio/EnsEMBL/Analysis/RunnableDB/ImportArrays.pm - Runs as a single job to 
collapse arrays of the same format into a non-redundant set of probes, storing probe records in the 
output DB and writing a non-reundant probe fasta file to be used in the alignment step.

ensembl-analysis/modules/Bio/EnsEMBL/Analysis/Runnable/ExonerateProbe.pm - Performs exonerate 
alignments and generates and filters ProbeFeatures given a max 
mismatch value.

ensembl-analysis/modules/Bio/EnsEMBL/Analysis/RunnableDB/ProbeAlign.pm - Runs the ExonerateProbe 
Runnable using genomic or transcript target sequence, and performs post processing and storage of 
ProbeFeatures, UnmappedObjects and DBEntries. Dependant on the target sequence the analysis run by 
{'fraction_non_alphanumeric': 0.11385314155942468, 'fraction_numerical': 0.012414837244511734, 'mean_word_length': 5.099261311172668, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3356694', 'n_tokens_mistral': 3750, 'n_tokens_neox': 3463, 'n_words': 1447}