<commit_before>"""===========================
Ancestral repeats pipeline
===========================

:Author: <NAME>
:Release: $Id: pipeline_ancestral_repeats.py 2876 2010-03-27 17:42:11Z andreas $
:Date: |today|
:Tags: Python

The ancestral repeats pipeline defines ancestral repeats for a pair of genomes
and computes rates for these.

This pipeline performs the following actions:

   * collect repeatmasker annotation from external
     databases. Currently implemented are:
      * UCSC
      * Ensembl
   * build pairwise genomic alignment from axt or maf files
   * define ancestral repeats
   * compute rates of ancestral repeats


Usage
=====

See :ref:`PipelineSettingUp` and :ref:`PipelineRunning` on general
information how to use CGAT pipelines.

Configuration
-------------

The pipeline expects a :term:`query` and :term:`target` genome. These
should be set in the general section.  For each genome there should
then be section on how to obtain the repeatmasker tracks. The default
configuration file gives an example.

Input
-----

The pipeline starts from an empty working directory. It will collect
the input data from directories specified in the configuration files.

The genomic alignment can both be build from :term:`axt` formatted
pairwise alignments and from :term:`maf` formatted multiple
alignments. However, the latter currently only works if the
:term:`query` genome is the reference species in the maf files.

This is a consequence of :file:`maf2Axt` requiring that the strand of
the reference species is always positive and I have not figured out
how to invert maf alignments.

.. note::
   ENSEMBL import is not thoroughly tested.
   :term:`maf` formatted import is not thoroughly tested.

Type::

   python pipeline_ancestral_repeats.py --help

for command line help.

Requirements
------------



Output
======

The pipeline builds the following files:

aligned_repeats.psl.gz<commit_msg>{AH} add documentation<commit_after>"""===========================
Ancestral repeats pipeline
===========================

:Author: <NAME>
:Release: $Id: pipeline_ancestral_repeats.py 2876 2010-03-27 17:42:11Z andreas $
:Date: |today|
:Tags: Python

The ancestral repeats pipeline defines ancestral repeats for a pair of genomes
and computes rates for these.

This pipeline performs the following actions:

   * collect repeatmasker annotation from external
     databases. Currently implemented are:
      * UCSC
      * Ensembl
   * build pairwise genomic alignment from axt or maf files
   * define ancestral repeats
   * compute rates of ancestral repeats

Usage
=====

See :ref:`PipelineSettingUp` and :ref:`PipelineRunning` on general
information how to use CGAT pipelines.

Configuration
-------------

The pipeline expects a :term:`query` and :term:`target` genome. These
should be set in the general section.  For each genome there should
then be section on how to obtain the repeatmasker tracks. The default
configuration file gives an example.

Input
-----

The pipeline starts from an empty working directory. It will collect
the input data from directories specified in the configuration files.

The genomic alignment can both be build from :term:`axt` formatted
pairwise alignments and from :term:`maf` formatted multiple
alignments. 

:term:`axt` formatted files (such as file:`chr1.hg19.mm10.net.axt.gz`)
are build by hierarchically from chains by selecting the
highest-scoring non-overlapping chains on top and then filling in the
gaps with lower scoring chains. A net is single-coverage for target
but not for query, unless it has been filtered to be single-coverage
on both target and query. Because it's single-coverage in the target,
it's no longer symmetrical. For ancestral repeat determination, use
:term:`axt` files that have been filtered to be single-coverage on
both query and target. By convention, the UCSC adds "rbest" to the net
filename in that case, such as: `hg19.panTro3.rbest.net.gz`.

:term:`maf` files currently only work if the :term:`query` genome is
the reference species in the maf files. This is a consequence of
:file:`maf2Axt` requiring that the strand of the reference species is
always positive and I have not figured out how to invert maf
alignments.

.. note::
   ENSEMBL import is not thoroughly tested.
   :term:`maf` formatted import is not thoroughly tested.

Type::

   python pipeline_ancestral_repeats.py --help

for command line help.

Requirements
------------



Output
======

The pipeline builds the following files:

aligned_repeats.psl.gz
{'fraction_non_alphanumeric': 0.09635359116022099, 'fraction_numerical': 0.01016574585635359, 'mean_word_length': 4.2935672514619885, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2968167', 'n_tokens_mistral': 1283, 'n_tokens_neox': 1159, 'n_words': 572}