<commit_before>High Availability (HA)
======================


High Availability in general terms means that we have 3 or more (up to 7) set up State Machines, 
each one can be used as the master.

This is an overview of how it works:

### Mongo
_Mongo_ is always started in [replicaset mode](http://docs.mongodb.org/manual/replication/):

 If not in HA, this will behave as if it ware a single mongodb and, in practical terms there is no difference with a regular setup.

### Ensure availability

There is a `ensure-availabiity` command for juju, it takes `-n` (minimum number of state machines) as an optional parameter, if its not provided it will default to **3**:

 This needs to be an odd number in order to prevent ties on the voting.
 
 The number can not be larger than seven (making the current possibilities: **3**, **5** and **7**) due to limitations of mongodb, which cannot have more than **7** replica set members.
 
 The number can be increased but not decreased: In the first case juju will bring up as many machines as necessary to meet the requirement, in the second nothing will happent since the rule tries to have _"at least that many"_
 
 At present there is no way to reduce the number of machines, you can kill by hand enough machines to reduce to a number you need, but this is risky and **not recommended**. If you kill less than half of the machines (half+1 remaining) running `ensure-availability` again will fix the set, if you kill more there is no way to fix it as there are not enough voting machines.
 
 Ensure availability will report what is going to be done after running it. 

### The API 

 There is an API server running on all State Machines, all of these talk to the master mongo db.
 
 The units will talk to any of the APIs.
 
 The first of the list to be tried is chosen at random, succesful connection's address will be stored to be used as first on subsequent connections.

### The peergrouper:
 
 The most relevant bits are in `worker/peergrouper/desired.go`

 It looks at the current state and decides what the peergroup members should look like, it continually tries to maintain those members

 The reason for its existence is that it can often take a while for mongo to allow a peer group change, so we can't change it directly in the API

 Its worker loop continally watches 

 1. The current set of state servers 
 2. The addresses of the current state servers 
 3. The status of the current mongo peergroup
 
It feeds all that information into `desiredPeerGroup`, which provides the peer group that we want to be and continually tries to set that peer group in mongo until it succeeds.
 
**NOTE:** There is one current situation that doesn't work currently which is that if you've only got one state server, you can't switch to another one <commit_msg>Added singleton and voting sections and corrected<commit_after>High Availability (HA)
======================


High Availability in general terms means that we have 3 or more (up to 7) 
State Machines, 
each one of which can be used as the master.

This is an overview of how it works:

### Mongo
_Mongo_ is always started in [replicaset mode](http://docs.mongodb.org/manual/replication/).

 If not in HA, this will behave as if it ware a single mongodb and, in practical 
terms there is no difference with a regular setup.

### Voting

A voting member of the replicaset is a one that has a saying on which member is master.

A non voting member its just a storage backup.

Currently we dont support non voting members; instead when a member is non voting it
mens that said State Server is going to be removed entirely

### Ensure availability

There is a `ensure-availabiity` command for juju, it takes `-n` (minimum number
 of state machines) as an optional parameter; if it's not provided it will 
default to 3.

 This needs to be an odd number in order to prevent ties on the voting.
 
 The number cannot be larger than seven (making the current possibilities: 3, 
5 and 7) due to limitations of mongodb, which cannot have more than 7
replica set members.
 
 Currently the number can be increased but not decreased (this is planned). 
In the first case juju will bring up as many machines as necessary to meet the 
requirement; in the second nothing will happen since the rule tries to have 
_"at least that many"_
 
 At present there is no way to reduce the number of machines, you can kill by 
hand enough machines to reduce to a number you need, but this is risky and 
**not recommended**. If you kill less than half of the machines (half+1
 remaining) running `ensure-availability` 
again will add more machines to replace the dead ones, if you kill more there 
is no way to recover as there are not enough voting machines.
 
 The EnsureAvailability API cal will report what is going to be done after 
running it. 

### The API 

 There is an API server running on all State Machines, these talk to all
the pers but queries and updates are addressed to the mongo master instance.
 
 Unit and machine agents connect to any of the API servers, by trying to connect
to all the addresses concurrently, but not simultaneously. It starts to try each
address in turn after a short delay. After a successful connection, the
connected address will be stored; it will be tried first when next connecting.

### The peergrouper worker:
 
 It looks at the current state and decides what the peergroup members should 
look like, it continually tries to maintain those members.

 The reason for its existence is that it can often take a while for mongo to 
allow a peer group change, so we can't change it directly in the 
EnsureAvailability API call

 Its worker loop continally watches 

 1. The current set of state servers 
 2. The addresses of the current state servers 
 3. The status of the current mongo peergroup
 
It feeds all that information into `desiredPeerGroup`, which provides the peer 
group that we want to be and continually tries to set that peer group in mongo 
until it succeeds.
 
**NOTE:** There is one current situation that doesn't work currently which is 
that if you've only got one state server, you can't switch to another one 

### The singleton Workers

The following workers require only a single instance to be running
at any one moment:

 * The environment provisioner
 * The firewaller
 * The charm revision updater
 * The state cleaner.
 * The transaction resumer
 * The minunits worker

When a machine agent connects to the state, it decides whether
it is on the same instance as the mongo master instance, and
if so, it runs the singleton workers; otherwise it doesn't run them.

Because we are using `mgo.Strong` consistency semantics,
it's guaranteed that our mongo connection will be dropped
when the master changes, which means that when the
master changes, the machine agent will reconnect to the
state and choose whether to run the singleton workers again.

It also means that we can never accidentally have two
singleton workers performing operations at the same time.
{'fraction_non_alphanumeric': 0.04448246364414029, 'fraction_numerical': 0.003136583974907328, 'mean_word_length': 4.050395968322534, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5685352', 'n_tokens_mistral': 1784, 'n_tokens_neox': 1655, 'n_words': 1169}