<commit_before>* Migration Goals and Priorities

All OHC API calls will be handled by Erlang-based web services that
store their data in PostgreSQL. We will deploy OHC in a configuration
that looks as much like OPC as possible. Acheiving these goals will
require multiple migrations. The priority list below will help us
shape the migrations to best meet business needs.

1. Performance and efficiency. Significantly reduce 95-percentile
   latency of API requests. Reduce RAM and CPU across both front and
   back end servers.
2. Tighter correspondence between OHC and OPC.
3. Remove couchdb from our infrastructure

** secondary goals
1. OPC migration tooling
2. OSC migration tooling

** Questions for all migrations
*** Do we want to use a different proxy/FE (not nginx) to make migration easier
*** Can we move the data in a downtime window
*** How much value is there in being able to guinea pig some orgs in new world
*** What happens if an org's data fails to migrate, or if we are only partially complete at end of window

* Migration 1: Nodes to MySQL (DONE)
* Migration 2: MySQL to PostgreSQL
** DONE Order hardware for a dedicated pgsql HA pair for oc_erchef data
** Assumptions
1. The community site runs on MySQL. It will not migrate to pgsql at
   this time. Community site will be part of a major reworking of the
   web properties and migration to pgsql will be part of the larger
   refactor.
** Questions
3. Can we do a full dump and load within an acceptable full outage
   window? Do we have other options? Batches of orgs? Users separate
   from nodes? Would a pre-load and UPDATE if modified time not
   matching be faster? Would customers prefer a scenario in which we
   snapshot and "restore" pg from snapshot and swap over so that OHC
   remains maximally available, but some data will be lost?
4. When will reporting be in its own db? Will it go straight into pg?
5. Does reporting data need to be migrated if we migrate before GA of
   reporting?
6. If reporting data needs to migrate, is there read-only data that
   could be moved ahead of time?
7. Should we upgrade to a sane user table schema before, after, or as
   a part of this migration? Where "sane" means looking much more like
   what is in OSC and where we upgrade hashed passwords to use bcrypt.
** Deploy postgres to preprod
*** cookbook development for pg
*** monitoring
** Deploy HA postgres to prod
*** test HA failover
** Establish postgres data backup
*** research backup options
Looks like zmanda has [[http://www.zmanda.com/postgres-backup.html][an offering]] so that might be easiest since we
are already using it for mysql.
*** implement backup
*** test recovery
** Write dump and load mysql to pg tool
*** Time dump/load time, is it reasonable for a full downtime outage?
** Test data dump/load in preprod
** Integrate pg client in oc-account
** Integrate pg client into oc-chef
** Integrate pg client into oc_erchef
** Integrate pg client into oc_reporting
** Community site?<commit_msg>Minor updates to ohc migration doc<commit_after>* Migration Goals and Priorities

All OHC API calls will be handled by Erlang-based web services that
store their data in PostgreSQL. We will deploy OHC in a configuration
that looks as much like OPC as possible. Acheiving these goals will
require multiple migrations. The priority list below will help us
shape the migrations to best meet business needs.

1. Performance and efficiency. Significantly reduce 95-percentile
   latency of API requests. Reduce RAM and CPU across both front and
   back end servers.
2. Tighter correspondence between OHC and OPC.
3. Remove couchdb from our infrastructure

** secondary goals
1. OPC migration tooling
2. OSC migration tooling

** Questions for all migrations
*** Do we want to use a different proxy/FE (not nginx) to make migration easier
*** Can we move the data in a downtime window
*** How much value is there in being able to guinea pig some orgs in new world
*** What happens if an org's data fails to migrate, or if we are only partially complete at end of window

* Migration 1: Nodes to MySQL (DONE)
* Migration 2: MySQL to PostgreSQL
** DONE Order hardware for a dedicated pgsql HA pair for oc_erchef data
** Assumptions
1. The community site runs on MySQL. It will not migrate to pgsql at
   this time. Community site will be part of a major reworking of the
   web properties and migration to pgsql will be part of the larger
   refactor.
** Questions
3. Can we do a full dump and load within an acceptable full outage
   window? Do we have other options? Batches of orgs? Users separate
   from nodes? Would a pre-load and UPDATE if modified time not
   matching be faster? Would customers prefer a scenario in which we
   snapshot and "restore" pg from snapshot and swap over so that OHC
   remains maximally available, but some data will be lost?
4. When will reporting be in its own db? Will it go straight into pg?
5. Does reporting data need to be migrated if we migrate before GA of
   reporting?
6. If reporting data needs to migrate, is there read-only data that
   could be moved ahead of time?
7. Should we upgrade to a sane user table schema before, after, or as
   a part of this migration? Where "sane" means looking much more like
   what is in OSC and where we upgrade hashed passwords to use bcrypt.
** Deploy postgres to preprod
*** cookbook development for pg
*** monitoring
** Deploy HA postgres to prod
*** test HA failover
** Establish postgres data backup
*** research backup options
Looks like zmanda has [[http://www.zmanda.com/postgres-backup.html][an offering]] so that might be easiest since we
are already using it for mysql.
*** implement backup
*** test recovery
** Write dump and load mysql to pg tool
*** Time dump/load time, is it reasonable for a full downtime outage?
** Test data dump/load in preprod
** Integrate pg client in oc-account
** Integrate pg client into oc-chef
** Integrate pg client into oc_erchef
{'fraction_non_alphanumeric': 0.04827586206896552, 'fraction_numerical': 0.005046257359125316, 'mean_word_length': 4.017721518987342, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5293782', 'n_tokens_mistral': 1566, 'n_tokens_neox': 1530, 'n_words': 957}