<commit_before>####
Part
####

part text 

*******
Chapter
*******

chapter text

Section
=======

section text

Subsection
----------

subsection text

Subsubsection
^^^^^^^^^^^^^

subsubsection text

Paragraph
'''''''''

paragraph text

.. = - ~ ` : ' " ^ _ * + # < >

#####
CAMHD
#####

This repository contains a Python module for interacting with data from the OOI
CamHD seafloor camera system stored in the raw data archive. It can be used to
obtain information about these remote files or retrieve individual frames
without downloading entire video files. The code here is currently under heavy
development, so the module is changing fast and often. We are considering moving
the code to a class-paradigm, so a lot may change in the coming months. Please
take this into consideration when developing code based on this module.

We are actively recruiting anyone interested in the CamHD data to participate in
the development of this code. Join up and contribute if you have time. Pull
requests greatly appreciated!

************
Installation
************

::

  $ hg clone https://bitbucket.org/tjcrone/camhd
  $ pip install camhd/dist/camhd-0.4.tar.gz

***********
Basic Usage
***********

**Write a frame to a single-frame AVI file**::

  >>> import camhd
  >>> filename = 'https://rawdata.oceanobservatories.org/files/RS03ASHS/PN03B/06-CAMHDA301/2016/11/13/CAMHDA301-20161113T000000Z.mov'
  >>> moov_atom = camhd.get_moov_atom(filename)
  >>> frame_count = camhd.get_frame_count(filename, moov_atom)
  >>> print(frame_count)
  >>> frame_number = 4976
  >>> camhd.write_frame(filename, frame_number, moov_atom)

The resulting AVI file can be converted to a TIFF, PNG, YUV, or another image or
movie format using ffmpeg. YUV conversions are lossless, as would be conversions to
any valid container format using a video stream copy. All CamHD ProRes encoded video
frames are key frames.

*Note: Obtaining the moov_atom first and passing it to any function is optional, but
it will greatly speed up calls to most functions. When multiple frames are to be
obtained from the same file, getting the moov_atom first is recommended.*

**Get information about the remote archive**::

  >>> (file_count, total_size) = camhd.get_stats()
  >>> print(file_count)
  >>> print(total_size)
  >>> file_list = camhd.get_file_list()
  >>> for filename in file_list:
  ...   print filename

*Note: Getting information about the repository can take several minutes, depending
on server response times, because every index file must be downloaded*

******************
Function Reference
******************

Archive Stats
=============

camhd.get_stats()
  Return the total number of MOV files and the total size of the MOV files
  (in TB) in the data archive. Returns an integer and a float.

camhd.get_file_list()
  Return a list of all MOV files in the data archive as fully-qualified URLs.
  Returns a list of strings.

Individual File Information
===========================

camhd.get_atom_sizes(filename)
  Return the sizes of the three top-level atoms in a remote file. Returns
  three integers.

camhd.get_chunk_count(filename[, moov_atom])
  Return the number of video chunks in a remote file. moov_atom should be a
  string containing raw packed binary data as returned by get_moov_atom().
  Returns an integer.

camhd.get_chunk_offsets(filename[, moov_atom])
  Return the offsets of all chunks in a remote file. Returns a list of
  integers.

camhd.get_frame_count(filename[, moov_atom])
  Return the number of frames in a remote file. Returns an integer.

camhd.get_frame_sizes(filename[, moov_atom])
  Return the sizes of all frames in a remote file. Returns a list of integers.

camhd.get_frame_offsets(filename[, moov_atom])
  Return the offsets of all frames in a remote file. Returns a list of
  integers.

Retrieve File Components
========================

camhd.get_moov_atom(filename)
  Retrieve the moov atom from a remote file. Returns a string containing raw
  packed binary data.

camhd.get_frame_data(filename, frame_number[, moov_atom])
  Retrieve the raw ProRes encoded frame data from a frame in a remote file.
  Returns a string containing raw packed binary data.

camhd.get_avi_file(frame_data)
  Adds an appropriately structured AVI header to frame_data. frame_data should
  be a string containing raw packed binary data as returned by
  get_frame_data(). Returns a string containing raw packed binary data.

Write Output File
=================

camhd.write_frame(filename, frame_number[, moov_atom])
  Writes a single-frame AVI file. The resulting AVI file can be converted to a
  TIFF, PNG, YUV, or another image or movie format using ffmpeg. YUV
  conversions are lossless, as would be conversions to any valid container
  format using a video stream copy. All CamHD ProRes encoded video frames are
  key frames.

Low-level Functions
===================

camhd.get_bytes(filename, byte_range)
  Retrieve a subset of bytes from a remote file. filename should be a fully
  qualified URL specifiying a remote CamHD Quicktime MOV file. byte_range
  should be a two-element list. Returns a string containing raw packed
  binary data.

camhd.get_integer(filename, byte_range)
  Return a 32-bit or 64-bit big-endian integer from a remote file.
  byte_range should be a two-element list specifying a 4-byte or 8-byte
  range.

Misc
====

camhd.version()
  Return the current version number of the module.

*******
License
*******

MIT License Copyright (c) 2016 <NAME>

******
Author
******

<NAME> (<EMAIL>)
<commit_msg>Update README.rst<commit_after>#####
CAMHD
#####

This repository contains a Python module for interacting with data from the OOI
CamHD seafloor camera system stored in the raw data archive. It can be used to
obtain information about these remote files or retrieve individual frames
without downloading entire video files. The code here is currently under heavy
development, so the module is changing fast and often. We are considering moving
the code to a class-paradigm, so a lot may change in the coming months. Please
take this into consideration when developing code based on this module.

We are actively recruiting anyone interested in the CamHD data to participate in
the development of this code. Join up and contribute if you have time. Pull
requests greatly appreciated!

************
Installation
************

::

  $ hg clone https://bitbucket.org/tjcrone/camhd
  $ pip install camhd/dist/camhd-0.4.tar.gz

***********
Basic Usage
***********

**Write a frame to a single-frame AVI file**::

  >>> import camhd
  >>> filename = 'https://rawdata.oceanobservatories.org/files/RS03ASHS/PN03B/06-CAMHDA301/2016/11/13/CAMHDA301-20161113T000000Z.mov'
  >>> moov_atom = camhd.get_moov_atom(filename)
  >>> frame_count = camhd.get_frame_count(filename, moov_atom)
  >>> print(frame_count)
  >>> frame_number = 4976
  >>> camhd.write_frame(filename, frame_number, moov_atom)

The resulting AVI file can be converted to a TIFF, PNG, YUV, or another image or
movie format using ffmpeg. YUV conversions are lossless, as would be conversions to
any valid container format using a video stream copy. All CamHD ProRes encoded video
frames are key frames.

*Note: Obtaining the moov_atom first and passing it to any function is optional, but
it will greatly speed up calls to most functions. When multiple frames are to be
obtained from the same file, getting the moov_atom first is recommended.*

**Get information about the remote archive**::

  >>> (file_count, total_size) = camhd.get_stats()
  >>> print(file_count)
  >>> print(total_size)
  >>> file_list = camhd.get_file_list()
  >>> for filename in file_list:
  ...   print filename

*Note: Getting information about the repository can take several minutes, depending
on server response times, because every index file must be downloaded*

******************
Function Reference
******************

Archive Stats
=============

camhd.get_stats()
  Return the total number of MOV files and the total size of the MOV files
  (in TB) in the data archive. Returns an integer and a float.

camhd.get_file_list()
  Return a list of all MOV files in the data archive as fully-qualified URLs.
  Returns a list of strings.

Individual File Information
===========================

camhd.get_atom_sizes(filename)
  Return the sizes of the three top-level atoms in a remote file. Returns
  three integers.

camhd.get_chunk_count(filename[, moov_atom])
  Return the number of video chunks in a remote file. moov_atom should be a
  string containing raw packed binary data as returned by get_moov_atom().
  Returns an integer.

camhd.get_chunk_offsets(filename[, moov_atom])
  Return the offsets of all chunks in a remote file. Returns a list of
  integers.

camhd.get_frame_count(filename[, moov_atom])
  Return the number of frames in a remote file. Returns an integer.

camhd.get_frame_sizes(filename[, moov_atom])
  Return the sizes of all frames in a remote file. Returns a list of integers.

camhd.get_frame_offsets(filename[, moov_atom])
  Return the offsets of all frames in a remote file. Returns a list of
  integers.

Retrieve File Components
========================

camhd.get_moov_atom(filename)
  Retrieve the moov atom from a remote file. Returns a string containing raw
  packed binary data.

camhd.get_frame_data(filename, frame_number[, moov_atom])
  Retrieve the raw ProRes encoded frame data from a frame in a remote file.
  Returns a string containing raw packed binary data.

camhd.get_avi_file(frame_data)
  Adds an appropriately structured AVI header to frame_data. frame_data should
  be a string containing raw packed binary data as returned by
  get_frame_data(). Returns a string containing raw packed binary data.

Write Output File
=================

camhd.write_frame(filename, frame_number[, moov_atom])
  Writes a single-frame AVI file. The resulting AVI file can be converted to a
  TIFF, PNG, YUV, or another image or movie format using ffmpeg. YUV
  conversions are lossless, as would be conversions to any valid container
  format using a video stream copy. All CamHD ProRes encoded video frames are
  key frames.

Low-level Functions
===================

camhd.get_bytes(filename, byte_range)
  Retrieve a subset of bytes from a remote file. filename should be a fully
  qualified URL specifiying a remote CamHD Quicktime MOV file. byte_range
  should be a two-element list. Returns a string containing raw packed
  binary data.

camhd.get_integer(filename, byte_range)
  Return a 32-bit or 64-bit big-endian integer from a remote file.
  byte_range should be a two-element list specifying a 4-byte or 8-byte
  range.

Misc
====

camhd.version()
  Return the current version number of the module.

*******
License
*******

MIT License Copyright (c) 2016 <NAME>

******
Author
******

<NAME> (<EMAIL>)

{'fraction_non_alphanumeric': 0.10160872781065089, 'fraction_numerical': 0.009245562130177515, 'mean_word_length': 4.460373548712771, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 82, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '554058', 'n_tokens_mistral': 3196, 'n_tokens_neox': 3026, 'n_words': 1351}