<commit_before>## Tower Clustering/HA Overview

Prior to 3.1 the Ansible Tower HA solution was not a true high-availability system. In 3.1 we have rewritten this system entirely with a new focus towards
a proper highly available clustered system. In 3.2 we have extended this further to allow grouping of clustered instances into different pools/queues.

* Each instance should be able to act as an entrypoint for UI and API Access.
  This should enable Tower administrators to use load balancers in front of as many instances as they wish
  and maintain good data visibility.
* Each instance should be able to join the Tower cluster and expand its ability to execute jobs.
* Provisioning new instance should be as simple as updating the `inventory` file and re-running the setup playbook
* Instances can be deprovisioned with a simple management commands
* Instances can be grouped into one or more Instance Groups to share resources for topical purposes.
* These instance groups should be assignable to certain resources:
  * Organizations
  * Inventories
  * Job Templates
  such that execution of jobs under those resources will favor particular queues

It's important to point out a few existing things:

* PostgreSQL is still a standalone instance and is not clustered. We also won't manage replica configuration or,
  if the user configures standby replicas, database failover.
* All instances should be reachable from all other instances and they should be able to reach the database. It's also important
  for the hosts to have a stable address and/or hostname (depending on how you configure the Tower host)
* RabbitMQ is the cornerstone of Tower's Clustering system. A lot of our configuration requirements and behavior is dictated
  by its needs. Thus we are pretty inflexible to customization beyond what our setup playbook allows. Each Tower instance has a
  deployment of RabbitMQ that will cluster with the other instances' RabbitMQ instances.
* Existing old-style HA deployments will be transitioned automatically to the new HA system during the upgrade process to 3.1.
* Manual projects will need to be synced to all instances by the customer

## Important Changes

* There is no concept of primary/secondary in the new Tower system. *All* systems are primary.
* Setup playbook changes to configure rabbitmq and give hints to the type of network the hosts are on.
* The `inventory` file for Tower deployments should be saved/persisted. If new instances are to be provisioned
  the passwords and configuration options as well as host names will need to be available to the installer.

## Concepts and Configuration

### Installation and the Inventory File
The current standalone instance configuration doesn't change for a 3.1+ deploy. The inventory file does change in some important ways:

* Since there is no primary/secondary configuration those inventory groups go away and are replaced with a
  single inventory group `tower`. The customer may, *optionally*, define other groups and group instances in those groups. These groups
  should be prefixed with `instance_group_`. Instances are not required to be in the `tower` group alongside other `instance_group_` groups, but one
  instance *must* be present in the `tower` group. Technically `tower` is a group like any other `instance_group_` group but it must always be present
  and if a specific group is not associated with a specific resource then job execution will always fall back to the `tower` group:
  ```
  [tower]
  hostA
  hostB
  hostC
  
  [instance_group_east]
  hostB
  hostC
  
  [instance_group_west]
  hostC
  hostD
  ```
  
  The `database` group remains for specifying an external postgres. If the database host is provisioned seperately this group should be empty
  ```
  [tower]
  hostA
  hostB
  hostC
  
  [database]
  hostDB
  ```

* It's common for customers to provision Tower instances externally but prefer to reference them by internal addressing. This is most significant
  for RabbitMQ clustering where the service isn't available at all on an external interface. For this purpose it is necessary to assign the internal
  address for RabbitMQ links as such:
  ```
  [tower]
  hostA rabbitmq_host=10.1.0.2
  hostB rabbitmq_host=10.1.0.3
  hostC rabbitmq_host=10.1.0.3
  ```
* The `redis_password` field is removed from `[all:vars]`
* There are various new fields for RabbitMQ:
  - `rabbitmq_port=5672` - RabbitMQ is installed on each instance and is not optional, it's also not possible to externalize. It is
    possible to configure what port it listens on and this setting controls that.
  - `rabbitmq_vhost=tower` - Tower configures a rabbitmq virtualhost to isolate itself. This controls that settings.
  - `rabbitmq_username=tower` and `rabbitmq_password=<PASSWORD>` - Each instance will be configured with these values and each instance's Tower
    instance will be configured with it also. This is similar to our other uses of usernames/passwords.
  - `rabbitmq_cookie=<somevalue>` - This value is unused in a standalone deployment but is critical for clustered deployments.
    This acts as the secret key that allows RabbitMQ cluster members to identify each other.
  - `rabbitmq_use_long_names` - RabbitMQ is pretty sensitive to what each instance is named. We are flexible enough to allow FQDNs
    (host01.example.com), short names (host01), or ip addresses (192.168.5.73). Depending on what is used to identify each host
    in the `inventory` file this value may need to be changed. For FQDNs and ip addresses this value needs to be `true`. For short
    names it should be `false`
  - `rabbitmq_enable_manager` - Setting this to `true` will expose the RabbitMQ management web console on each instance.

The most important field to point out for variability is `rabbitmq_use_long_name`. That's something we can't detect or provide a reasonable
default for so it's important to point out when it needs to be changed. If instances are provisioned to where they reference other instances
internally and not on external addressess then `rabbitmq_use_long_name` semantics should follow the internal addressing (aka `rabbitmq_host`.

Other than `rabbitmq_use_long_name` the defaults are pretty reasonable:
```
rabbitmq_port=5672
rabbitmq_vhost=tower
rabbitmq_username=tower
rabbitmq_password=''
rabbitmq_cookie=<PASSWORD>

# Needs to be true for fqdns and ip addresses
rabbitmq_use_long_name=false
rabbitmq_enable_manager=false
```

### Security Isolated Rampart Groups

In Tower versions 3.2+ customers may optionally define isolated groups
inside security-restricted networking zones to run jobs and ad hoc commands from.
Instances in these groups will _not_ have a full install of Tower, but will have a minimal
set of utilities used to run jobs. Isolated groups must be specified
in the inventory file prefixed with `isolated_group_`. An example inventory
file is shown below.

```
[tower]
towerA
towerB
towerC

[instance_group_security]
towerB
towerC

[isolated_group_govcloud]
isolatedA
isolatedB

[isolated_group_govcloud:vars]
controller=security
```

In the isolated rampart model, "controller" instances interact with "isolated"
instances via a series of Ansible playbooks over SSH.  As such, all isolated instances
must be preconfigured by the installer with passwordless SSH access from any potential
controller instances.  In the example above, the `isolatedA` and `isolatedB` hosts
must be reachable from `towerB` and `towerC` hosts via `ssh
awx@<isolated-hostname>` (meaning, `authorized_keys` must be pre-distributed to
the `isolatedA` and `isolatedB` hosts).

When a job is scheduled to run on an "isolated" instance:

*  The "controller" instance compiles metadata required to run the job and copies
   it to the "isolated" instance via `rsync` (any related project or inventory
   updates are run on the controller instance). This metadata includes:

   - the entire SCM checkout directory for the project
   - a static inventory file
   - pexpect passwords
   - environment variables
   - the `ansible`/`ansible-playbook` command invocation, i.e., 
     `bwrap ... ansible-playbook -i /path/to/inventory /path/to/playbook.yml -e ...`

* Once the metadata has been rsynced to the isolated host, the "controller
  instance" starts a process on the "isolated" instance which consumes the
  metadata and starts running `ansible`/`ansible-playbook`.  As the playbook
  runs, job artifacts (such as stdout and job events) are written to disk on
  the "isolated" instance.

* While the job runs on the "isolated" instance, the "controller" instance
  periodically copies job artifacts (stdout and job events) from the "isolated"
  instance using `rsync`.  It consumes these until the job finishes running on the
  "isolated" instance.

Isolated groups are architected such that they may exist inside of a VPC
with security rules that _only_ permit the instances in its `controller`
group to access them; only ingress SSH traffic from "controller" instances to
"isolated" instances is required.

Recommendations for system configuration with isolated groups:
 - Do not put any isolated instances inside the `tower` group or other
   ordinary instance groups.
 - Define the `controller` variable as either a group var or as a hostvar
   on all the instances in the isolated group. Please _do not_ allow
   isolated instances in the same group have a different value for this
   variable - the behavior in this case can not be predicted.
 - Do not put an isolated instance in more than 1 isolated group.


### Provisioning and Deprovisioning Instances and Groups

* Provisioning
Provisioning Instances after installation is supported by updating the `inventory` file and re-running the setup playbook. It's important that this file
contain all passwords and information used when installing the cluster or other instances may be reconfigured (This could be intentional)

* Deprovisioning
Tower does not automatically de-provision instances since we can't distinguish between an instance that was taken offline intentionally or due to failure.
Instead the procedure for deprovisioning an instance is to shut it down (or stop the `ansible-tower-service`) and run the Tower deprovision command:

```
$ tower-manage deprovision-node <nodename>
```

* Removing/Deprovisioning Instance Groups
Tower does not automatically de-provision or remove instance groups, even though re-provisioning will often cause these to be unused. They may still
show up in api endpoints and stats monitoring. These groups can be removed with the following command:
<commit_msg>document namings you should never use<commit_after>## Tower Clustering/HA Overview

Prior to 3.1 the Ansible Tower HA solution was not a true high-availability system. In 3.1 we have rewritten this system entirely with a new focus towards
a proper highly available clustered system. In 3.2 we have extended this further to allow grouping of clustered instances into different pools/queues.

* Each instance should be able to act as an entrypoint for UI and API Access.
  This should enable Tower administrators to use load balancers in front of as many instances as they wish
  and maintain good data visibility.
* Each instance should be able to join the Tower cluster and expand its ability to execute jobs.
* Provisioning new instance should be as simple as updating the `inventory` file and re-running the setup playbook
* Instances can be deprovisioned with a simple management commands
* Instances can be grouped into one or more Instance Groups to share resources for topical purposes.
* These instance groups should be assignable to certain resources:
  * Organizations
  * Inventories
  * Job Templates
  such that execution of jobs under those resources will favor particular queues

It's important to point out a few existing things:

* PostgreSQL is still a standalone instance and is not clustered. We also won't manage replica configuration or,
  if the user configures standby replicas, database failover.
* All instances should be reachable from all other instances and they should be able to reach the database. It's also important
  for the hosts to have a stable address and/or hostname (depending on how you configure the Tower host)
* RabbitMQ is the cornerstone of Tower's Clustering system. A lot of our configuration requirements and behavior is dictated
  by its needs. Thus we are pretty inflexible to customization beyond what our setup playbook allows. Each Tower instance has a
  deployment of RabbitMQ that will cluster with the other instances' RabbitMQ instances.
* Existing old-style HA deployments will be transitioned automatically to the new HA system during the upgrade process to 3.1.
* Manual projects will need to be synced to all instances by the customer

## Important Changes

* There is no concept of primary/secondary in the new Tower system. *All* systems are primary.
* Setup playbook changes to configure rabbitmq and give hints to the type of network the hosts are on.
* The `inventory` file for Tower deployments should be saved/persisted. If new instances are to be provisioned
  the passwords and configuration options as well as host names will need to be available to the installer.

## Concepts and Configuration

### Installation and the Inventory File
The current standalone instance configuration doesn't change for a 3.1+ deploy. The inventory file does change in some important ways:

* Since there is no primary/secondary configuration those inventory groups go away and are replaced with a
  single inventory group `tower`. The customer may, *optionally*, define other groups and group instances in those groups. These groups
  should be prefixed with `instance_group_`. Instances are not required to be in the `tower` group alongside other `instance_group_` groups, but one
  instance *must* be present in the `tower` group. Technically `tower` is a group like any other `instance_group_` group but it must always be present
  and if a specific group is not associated with a specific resource then job execution will always fall back to the `tower` group:
  ```
  [tower]
  hostA
  hostB
  hostC
  
  [instance_group_east]
  hostB
  hostC
  
  [instance_group_west]
  hostC
  hostD
  ```
  
  The `database` group remains for specifying an external postgres. If the database host is provisioned seperately this group should be empty
  ```
  [tower]
  hostA
  hostB
  hostC
  
  [database]
  hostDB
  ```

* It's common for customers to provision Tower instances externally but prefer to reference them by internal addressing. This is most significant
  for RabbitMQ clustering where the service isn't available at all on an external interface. For this purpose it is necessary to assign the internal
  address for RabbitMQ links as such:
  ```
  [tower]
  hostA rabbitmq_host=10.1.0.2
  hostB rabbitmq_host=10.1.0.3
  hostC rabbitmq_host=10.1.0.3
  ```
* The `redis_password` field is removed from `[all:vars]`
* There are various new fields for RabbitMQ:
  - `rabbitmq_port=5672` - RabbitMQ is installed on each instance and is not optional, it's also not possible to externalize. It is
    possible to configure what port it listens on and this setting controls that.
  - `rabbitmq_vhost=tower` - Tower configures a rabbitmq virtualhost to isolate itself. This controls that settings.
  - `rabbitmq_username=tower` and `rabbitmq_password=<PASSWORD>` - Each instance will be configured with these values and each instance's Tower
    instance will be configured with it also. This is similar to our other uses of usernames/passwords.
  - `rabbitmq_cookie=<somevalue>` - This value is unused in a standalone deployment but is critical for clustered deployments.
    This acts as the secret key that allows RabbitMQ cluster members to identify each other.
  - `rabbitmq_use_long_names` - RabbitMQ is pretty sensitive to what each instance is named. We are flexible enough to allow FQDNs
    (host01.example.com), short names (host01), or ip addresses (192.168.5.73). Depending on what is used to identify each host
    in the `inventory` file this value may need to be changed. For FQDNs and ip addresses this value needs to be `true`. For short
    names it should be `false`
  - `rabbitmq_enable_manager` - Setting this to `true` will expose the RabbitMQ management web console on each instance.

The most important field to point out for variability is `rabbitmq_use_long_name`. That's something we can't detect or provide a reasonable
default for so it's important to point out when it needs to be changed. If instances are provisioned to where they reference other instances
internally and not on external addressess then `rabbitmq_use_long_name` semantics should follow the internal addressing (aka `rabbitmq_host`.

Other than `rabbitmq_use_long_name` the defaults are pretty reasonable:
```
rabbitmq_port=5672
rabbitmq_vhost=tower
rabbitmq_username=tower
rabbitmq_password=''
rabbitmq_cookie=<PASSWORD>

# Needs to be true for fqdns and ip addresses
rabbitmq_use_long_name=false
rabbitmq_enable_manager=false
```

Recommendations and constraints:
 - Do not create a group named `instance_group_tower`
 - Do not name any instance the same as a group name

### Security Isolated Rampart Groups

In Tower versions 3.2+ customers may optionally define isolated groups
inside security-restricted networking zones to run jobs and ad hoc commands from.
Instances in these groups will _not_ have a full install of Tower, but will have a minimal
set of utilities used to run jobs. Isolated groups must be specified
in the inventory file prefixed with `isolated_group_`. An example inventory
file is shown below.

```
[tower]
towerA
towerB
towerC

[instance_group_security]
towerB
towerC

[isolated_group_govcloud]
isolatedA
isolatedB

[isolated_group_govcloud:vars]
controller=security
```

In the isolated rampart model, "controller" instances interact with "isolated"
instances via a series of Ansible playbooks over SSH.  As such, all isolated instances
must be preconfigured by the installer with passwordless SSH access from any potential
controller instances.  In the example above, the `isolatedA` and `isolatedB` hosts
must be reachable from `towerB` and `towerC` hosts via `ssh
awx@<isolated-hostname>` (meaning, `authorized_keys` must be pre-distributed to
the `isolatedA` and `isolatedB` hosts).

When a job is scheduled to run on an "isolated" instance:

*  The "controller" instance compiles metadata required to run the job and copies
   it to the "isolated" instance via `rsync` (any related project or inventory
   updates are run on the controller instance). This metadata includes:

   - the entire SCM checkout directory for the project
   - a static inventory file
   - pexpect passwords
   - environment variables
   - the `ansible`/`ansible-playbook` command invocation, i.e., 
     `bwrap ... ansible-playbook -i /path/to/inventory /path/to/playbook.yml -e ...`

* Once the metadata has been rsynced to the isolated host, the "controller
  instance" starts a process on the "isolated" instance which consumes the
  metadata and starts running `ansible`/`ansible-playbook`.  As the playbook
  runs, job artifacts (such as stdout and job events) are written to disk on
  the "isolated" instance.

* While the job runs on the "isolated" instance, the "controller" instance
  periodically copies job artifacts (stdout and job events) from the "isolated"
  instance using `rsync`.  It consumes these until the job finishes running on the
  "isolated" instance.

Isolated groups are architected such that they may exist inside of a VPC
with security rules that _only_ permit the instances in its `controller`
group to access them; only ingress SSH traffic from "controller" instances to
"isolated" instances is required.

Recommendations for system configuration with isolated groups:
 - Do not create a group named `isolated_group_tower`
 - Do not put any isolated instances inside the `tower` group or other
   ordinary instance groups.
 - Define the `controller` variable as either a group var or as a hostvar
   on all the instances in the isolated group. Please _do not_ allow
   isolated instances in the same group have a different value for this
   variable - the behavior in this case can not be predicted.
 - Do not put an isolated instance in more than 1 isolated group.


### Provisioning and Deprovisioning Instances and Groups

* Provisioning
Provisioning Instances after installation is supported by updating the `inventory` file and re-running the setup playbook. It's important that this file
contain all passwords and information used when installing the cluster or other instances may be reconfigured (This could be intentional)

* Deprovisioning
Tower does not automatically de-provision instances since we can't distinguish between an instance that was taken offline intentionally or due to failure.
Instead the procedure for deprovisioning an instance is to shut it down (or stop the `ansible-tower-service`) and run the Tower deprovision command:

```
$ tower-manage deprovision-node <nodename>
```

* Removing/Deprovisioning Instance Groups
Tower does not automatically de-provision or remove instance groups, even though re-provisioning will often cause these to be unused. They may still
show up in api endpoints and stats monitoring. These groups can be removed with the following command:

{'fraction_non_alphanumeric': 0.04603939549899406, 'fraction_numerical': 0.004585224348477051, 'mean_word_length': 4.488957370313303, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4395488', 'n_tokens_mistral': 5735, 'n_tokens_neox': 5376, 'n_words': 2985}