<commit_before>WAL-E Disaster Recovery
=======================

.. contents:: Table of Contents

Introduction
------------

WAL-E is a program designed to perform continuous archiving of PostgreSQL
WAL files and manage the use of pg_start_backup and pg_stop_backup.

To correspond on using WAL-E or to collaborate on its development, do
not hesitate to send mail to the mailing list at
<EMAIL>.  Github issues are also currently being used
to track known problems, so please feel free to submit those.


Primary Commands and Concepts
-----------------------------

WAL-E has four critical operators:

* backup-fetch
* backup-push
* wal-fetch
* wal-push

Of these, the "push" operators send things to storage and "fetch"
operators get things from storage.  "wal" operators send/get write ahead
log, and "backup" send/get a hot backup of the base database that WAL
segments can be applied to.

All of these operators work in a context of three important
environment-variable based settings:

* AWS_ACCESS_KEY_ID or WABS_ACCOUNT_NAME
* AWS_SECRET_ACCESS_KEY or WABS_ACCESS_KEY
* WALE_S3_PREFIX or WALE_WABS_PREFIX

For Swift the following environment variables are needed:

* SWIFT_AUTHURL
* SWIFT_TENANT
* SWIFT_USER
* SWIFT_PASSWORD
* WALE_SWIFT_PREFIX

With the exception of AWS_SECRET_ACCESS_KEY and WABS_ACCESS_KEY, all
of these can be specified as arguments as well.  The AWS_* variables
are the standard access-control keying system provided by Amazon,
where the WABS_* are the standard access credentials defined by
Windows Azure.

The WALE_S3_PREFIX, WALE_WABS_PREFIX and WALE_SWIFT_PREFIX (_PREFIX)
variables can be thought of as a context whereby this program operates
on a single database cluster at a time.  Generally, for any one
database the _PREFIX will be the same between all four operators.
This context-driven approach attempts to help users avoid errors such
as one database overwriting the WAL segments of another, as long as
the _PREFIX is set uniquely for each database. Use whichever variable
is appropriate for the store you are using.

.. IMPORTANT::
   Ensure that all servers have different _PREFIXes set.
   Reuse of a value between two servers will likely cause unrecoverable
   backups.


Dependencies
------------

* lzop
* psql (>= 8.4)
* pv

This software is most frequently used with Python 2.6+.  It will
probably never support Python 2.5 or below because of the much more
useful timeout semantics with sockets in Python 2.6+.  Still, if you
feel strongly about supporting Python 2.5, please send mail to the
mailing list (see the `Introduction`_).

This software also has Python dependencies; installing with setup.py
will attempt to resolve them:

* gevent>=0.13.1
* boto>=2.6.0
* azure==0.7.0
* argparse, if not on Python 2.7

Backend Blob Store
------------------

The storage backend is determined by the defined _PREFIX. Prefixes with the
scheme ``s3`` will be directed towards S3, those with the scheme ``wabs`` will
be directed towards Windows Azure Blob Service, and those with the scheme
``swift`` will be directed towards an OpenStack Swift installation.<commit_msg>Update dependency documentation in README<commit_after>WAL-E Disaster Recovery
=======================

.. contents:: Table of Contents

Introduction
------------

WAL-E is a program designed to perform continuous archiving of PostgreSQL
WAL files and manage the use of pg_start_backup and pg_stop_backup.

To correspond on using WAL-E or to collaborate on its development, do
not hesitate to send mail to the mailing list at
<EMAIL>.  Github issues are also currently being used
to track known problems, so please feel free to submit those.


Primary Commands and Concepts
-----------------------------

WAL-E has four critical operators:

* backup-fetch
* backup-push
* wal-fetch
* wal-push

Of these, the "push" operators send things to storage and "fetch"
operators get things from storage.  "wal" operators send/get write ahead
log, and "backup" send/get a hot backup of the base database that WAL
segments can be applied to.

All of these operators work in a context of three important
environment-variable based settings:

* AWS_ACCESS_KEY_ID or WABS_ACCOUNT_NAME
* AWS_SECRET_ACCESS_KEY or WABS_ACCESS_KEY
* WALE_S3_PREFIX or WALE_WABS_PREFIX

For Swift the following environment variables are needed:

* SWIFT_AUTHURL
* SWIFT_TENANT
* SWIFT_USER
* SWIFT_PASSWORD
* WALE_SWIFT_PREFIX

With the exception of AWS_SECRET_ACCESS_KEY and WABS_ACCESS_KEY, all
of these can be specified as arguments as well.  The AWS_* variables
are the standard access-control keying system provided by Amazon,
where the WABS_* are the standard access credentials defined by
Windows Azure.

The WALE_S3_PREFIX, WALE_WABS_PREFIX and WALE_SWIFT_PREFIX (_PREFIX)
variables can be thought of as a context whereby this program operates
on a single database cluster at a time.  Generally, for any one
database the _PREFIX will be the same between all four operators.
This context-driven approach attempts to help users avoid errors such
as one database overwriting the WAL segments of another, as long as
the _PREFIX is set uniquely for each database. Use whichever variable
is appropriate for the store you are using.

.. IMPORTANT::
   Ensure that all servers have different _PREFIXes set.
   Reuse of a value between two servers will likely cause unrecoverable
   backups.


Dependencies
------------

* lzop
* psql (>= 8.4)
* pv

This software is most frequently used with Python 2.6+.  It will
probably never support Python 2.5 or below because of the much more
useful timeout semantics with sockets in Python 2.6+.  Still, if you
feel strongly about supporting Python 2.5, please send mail to the
mailing list (see the `Introduction`_).

This software also has Python dependencies; installing with setup.py
will attempt to resolve them:

* gevent>=0.13.1
* boto>=2.6.0
* azure>=0.7.0
* python-swiftclient>=1.8.0
* python-keystoneclient>=0.4.2
* argparse, if not on Python 2.7

It is possible to use WAL-E without the dependencies of back-end
storage one does not use installed: the imports for those are only
performed if the storage configuration demands their use.

Backend Blob Store
------------------

The storage backend is determined by the defined _PREFIX. Prefixes with the
scheme ``s3`` will be directed towards S3, those with the scheme ``wabs`` will
be directed towards Windows Azure Blob Service, and those with the scheme
``swift`` will be directed towards an OpenStack Swift installation.
{'fraction_non_alphanumeric': 0.07292307692307692, 'fraction_numerical': 0.008923076923076922, 'mean_word_length': 4.094827586206897, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '850349', 'n_tokens_mistral': 1825, 'n_tokens_neox': 1762, 'n_words': 894}