<commit_before>
=============================================
Accessing, Building, and Querying Annotations
=============================================

Purpose
^^^^^^^

This tutorial teaches you the finer points of constructing
annotation databases, mapping them onto associated sequence
databases, and saving them in :mod:`worldbase`.  You should 
understand Pygr sequences (see :doc:`sequence`).  While not
required, you may find it helpful to view the :doc:`db_basic`
tutorial, to get comfortable with Pygr's different database types.

Pygr Sequence Annotations
^^^^^^^^^^^^^^^^^^^^^^^^^

*Annotation* -- information bound to specific intervals of a genome
or sequence -- is an essential concept in bioinformatics.  We would like to
be able to store and query annotations naturally as part of working with
sequence datasets of any size, from single sequences to multigenome
alignments.  Pygr makes this fairly easy.

An annotation is an interval (i.e. it has length, and can be sliced,
or negated to get the opposite strand) that has bound attributes giving
biological information about the annotation.  It acts like a little coordinate
system, i.e. ``annotation[0]`` is the first position in the annotation;
``annotation[-10:]`` is the last ten positions of the annotation etc.
Any subslice of the annotation gives its coordinates and orientation
relative to the original annotation.  

Let's set up a basic example, by  opening a sequence database to
annotate, and a ``MySliceInfo`` class that will store the basic
coordinate attributes (``id``, ``start``, ``stop``, ``orientation``
required by a default Pygr :class:`annotation.AnnotationDB` database::
 
  >>> from slice_pickle_obj import MySliceInfo, MyFunkySliceInfo
  >>> from pygr import seqdb, annotation, worldbase

  >>> dna_db = seqdb.SequenceFileDB('../tests/data/hbb1_mouse.fa')
  >>> seq_id = 'gi|171854975|dbj|AB364477.1|'
  >>> seq = dna_db[seq_id]

Constructing an annotation database using regular Python objects
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To start simple, let's just save a couple annotation intervals
to our database using its :meth:`annotation.AnnotationDB.new_annotation()`
method::

  >>> annodb = annotation.AnnotationDB({}, dna_db)

  >>> slice1 = MySliceInfo(seq_id, 0, 50, +1)
  >>> slice2 = MySliceInfo(seq_id, 300, 400, -1)

  >>> annot1 = annodb.new_annotation('A', slice1)
  >>> annot2 = annodb.new_annotation('B', slice2)

  >>> for k in annodb:
  ...     print repr(annodb[k]), repr(annodb[k].sequence)
  annotA[0:50] gi|171854975|dbj|AB364477.1|[0:50]
  annotB[0:100] -gi|171854975|dbj|AB364477.1|[300:400]

An annotation object or slice is associated with the corresponding
sequence interval of the sequence to which it is bound.  To obtain that
sequence interval, simply request the annotation's :attr:`sequence`
attribute.  Use this if you want to get its sequence string via ``str``,
for example.

While an annotation behaves like a sequence in most respects,
usually it will *not* provide a string value::

  >>> str(annodb['A'])
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/Users/leec/projects/pygr/pygr/sequence.py", line 483, in __str__
       return self.path.strslice(self.start,self.stop)
     File "/Users/leec/projects/pygr/pygr/annotation.py", line 61, in strslice
       Use its sequence attribute to get a sequence object representing this interval.''')
   ValueError: this is an annotation, and you cannot get a sequence string from it.
   Use its sequence attribute to get a sequence object representing this interval.

One exception to this rule: :class:`annotation.TranslationAnnot`
annotations, which represent the protein translation of an underlying
nucleotide sequence interval.  
  
Modifying annotationType
^^^^^^^^^^^^^^^^^^^^^^^^

  >>> annodb = annotation.AnnotationDB({}, dna_db, annotationType='foo:')

  >>> slice1 = MySliceInfo(seq_id, 0, 50, +1)
  >>> slice2 = MySliceInfo(seq_id, 300, 400, -1)

  >>> annot1 = annodb.new_annotation('A', slice1)
  >>> annot2 = annodb.new_annotation('B', slice2)

  >>> for k in annodb:
  ...    print repr(annodb[k])<commit_msg>more little changes to annotation2.rst<commit_after>
=============================================
Accessing, Building, and Querying Annotations
=============================================

Purpose
^^^^^^^

This tutorial teaches you the finer points of constructing
annotation databases, mapping them onto associated sequence
databases, and saving them in :mod:`worldbase`.  You should 
understand Pygr sequences (see :doc:`sequence`).  While not
required, you may find it helpful to view the :doc:`db_basic`
tutorial, to get comfortable with Pygr's different database types.

Pygr Sequence Annotations
^^^^^^^^^^^^^^^^^^^^^^^^^

*Annotation* -- information bound to specific intervals of a genome
or sequence -- is an essential concept in bioinformatics.  We would like to
be able to store and query annotations naturally as part of working with
sequence datasets of any size, from single sequences to multigenome
alignments.  Pygr makes this fairly easy.

An annotation is an interval (i.e. it has length, and can be sliced,
or negated to get the opposite strand) that has bound attributes giving
biological information about the annotation.  It acts like a little coordinate
system, i.e. ``annotation[0]`` is the first position in the annotation;
``annotation[-10:]`` is the last ten positions of the annotation etc.
Any subslice of the annotation gives its coordinates and orientation
relative to the original annotation.  

Let's set up a basic example, by  opening a sequence database to
annotate, and a ``MySliceInfo`` class that will store the basic
coordinate attributes (``id``, ``start``, ``stop``, ``orientation``
required by a default Pygr :class:`annotation.AnnotationDB` database::
 
  >>> from slice_pickle_obj import MySliceInfo, MyFunkySliceInfo
  >>> from pygr import seqdb, annotation, worldbase

  >>> dna_db = seqdb.SequenceFileDB('../tests/data/hbb1_mouse.fa')
  >>> seq_id = 'gi|171854975|dbj|AB364477.1|'

Constructing an annotation database using regular Python objects
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To start simple, let's just save a couple annotation intervals
to our database using its :meth:`annotation.AnnotationDB.new_annotation()`
method::

  >>> annodb = annotation.AnnotationDB({}, dna_db)

  >>> slice1 = MySliceInfo(seq_id, 0, 50, +1)
  >>> slice2 = MySliceInfo(seq_id, 300, 400, -1)

  >>> annot1 = annodb.new_annotation('A', slice1)
  >>> annot2 = annodb.new_annotation('B', slice2)

An annotation object or slice is associated with the corresponding
sequence interval of the sequence to which it is bound.  To obtain that
sequence interval, simply request the annotation's :attr:`sequence`
attribute.  Use this if you want to get its sequence string via ``str``,
for example::

  >>> for k in annodb:
  ...     print repr(annodb[k]), repr(annodb[k].sequence)
  annotA[0:50] gi|171854975|dbj|AB364477.1|[0:50]
  annotB[0:100] -gi|171854975|dbj|AB364477.1|[300:400]

As a general, an object from a Pygr database have
two attributes that identify it as belonging to its database:
``id`` gives its unique identifier (primary key) in that database,
and ``db`` points to the annotation database object itself::

  >>> a = annodb['A']
  >>> a.id
  'A'
  >>> a.db is annodb
  True

While an annotation behaves like a sequence in most respects,
usually it will *not* provide a string value::

  >>> str(a)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/Users/leec/projects/pygr/pygr/sequence.py", line 483, in __str__
       return self.path.strslice(self.start,self.stop)
     File "/Users/leec/projects/pygr/pygr/annotation.py", line 61, in strslice
       Use its sequence attribute to get a sequence object representing this interval.''')
   ValueError: this is an annotation, and you cannot get a sequence string from it.
   Use its sequence attribute to get a sequence object representing this interval.

One exception to this rule: :class:`annotation.TranslationAnnot`
annotations, which represent the protein translation of an underlying
nucleotide sequence interval.  
  
Modifying annotationType
^^^^^^^^^^^^^^^^^^^^^^^^

An annotation object or slice always has an
:attr:`annotation.AnnotationDB.annotationType` 
attribute giving a string identifier for
its annotation type.  You can set this by passing an argument::

  >>> annodb = annotation.AnnotationDB({}, dna_db, annotationType='foo:')

  >>> slice1 = MySliceInfo(seq_id, 0, 50, +1)
  >>> slice2 = MySliceInfo(seq_id, 300, 400, -1)

  >>> annot1 = annodb.new_annotation('A', slice1)
  >>> annot2 = annodb.new_annotation('B', slice2)

  >>> for k in annodb:
  ...    print repr(annodb[k])
{'fraction_non_alphanumeric': 0.14004524886877828, 'fraction_numerical': 0.024547511312217194, 'mean_word_length': 4.894, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 121, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3203478', 'n_tokens_mistral': 2725, 'n_tokens_neox': 2433, 'n_words': 998}