<commit_before>This file contains notes for upgrades when special instructions are necessary.
In general, simply doing:

# Get the latest code
git pull

# Gather any new static files
manage.py collectstatic

(enter yes, when prompted)

# Migrate the database
manage.py migrate

# Restart celery
sudo service celeryd restart

# Restart apache's code
touch apache/django.wsgi

is all that's necessary. In the case that special instructions are needed, we
will attempt to put them here, with the most recent at the top.

We welcome a conversion of these notes to a better process using Fabric.

----------------------------


2014-10-23:
 Overview:
 - This is a very large upgrade and brings the concept of the "Docket" into
   CourtListener. After this point, we will have both opinions and oral
   arguments for the first time, with the result being that we need some kind
   of container that we can hold them in. A new class has been added to
   search.models called "Docket" to hold this kind of information.

 - When upgrading to this version, everything will completely break until your
   database is updated. I recommend enabling the maintenance mode URL if you
   wish to avoid chaos for your users.

 - The database upgrade for this version will take some time, as we are moving
   a number of fields from one database table to another. A re-index is also
   required, but, if you have ample disk space, this will not cause downtime.

  Deployment:
  - Install new dependencies:
    - sudo pip install --allow-external eyeD3 --allow-unverified eyeD3 eyeD3==0.7.4
    - sudo pip install datadiff==1.1.5
    - avconv: sudo apt-get install libav-tools
  - Install the seal-rookery:
    - sudo git clone https://github.com/freelawproject/seal-rookery /usr/local/seal_rookery
    - sudo ln -s /usr/local/seal_rookery /usr/lib/python2.7/dist-packages/seal_rookery

  - Disable any current scrapers using killall, and disable cron jobs from
    kicking in again.
  - Flush any background tasks with countdowns in the celery queue by following
    the notes here: https://stackoverflow.com/questions/26535719/
  - Stop Celery
  - Pull the code and the maintenance mode will be enabled.
  - Restart apache to enable the new code and clear its caches:
        - cl; sudo touch apache/django.wsgi
  - Database changes
        - manage.py migrate search --delete-ghost-migrations
        - manage.py migrate
  - Add a new core to Solr (paths below may need changing to correspond with
    the location of your current configurations):

        1. Create a new configuration directory based on our current one:

           % sudo cp -r /usr/local/solr/example/solr/collection1 /usr/local/solr/example/solr/audio
           % source /etc/courtlistener
           % sudo ln -s -f $INSTALL_ROOT/Solr/conf/audio_schema.xml /usr/local/solr/example/solr/audio/conf/schema.xml

        2. Create the new core in Python:

            from alert.lib.solr_core_admin import create_solr_core
            from alert import settings
            import os
            create_solr_core(
               core_name='audio',
               data_dir=os.path.join(settings.INSTALL_ROOT, 'Solr', 'data_audio'),
               schema=os.path.join(settings.INSTALL_ROOT, 'Solr', 'conf', 'audio_schema.xml'),
               instance_dir='/usr/local/solr/example/solr/audio',
            )

  - Turn maintenance mode back off.
  - Re-start celery
  - Re-enable scrapers and kick off the oral argument scrapers
  - Flip the has_oral_argument_scraper flag to true on all correct courts.
  - absolute_urls have changed throughout, necessitating a complete re-index
    for links in search results to work without redirects. We can re-index in
    a background core and then swap it in using the following commands:

        >>> create_solr_core(
            core_name='swap_core',
            data_dir=os.path.join(settings.INSTALL_ROOT, 'Solr', 'data_swap'),
            schema=os.path.join(settings.INSTALL_ROOT, 'Solr', 'conf', 'schema.xml'),
        )

        % screen
        % manage.py cl_update_index --update --everything --optimize --solr-url http://127.0.0.1:8983/solr/swap_core --type opinions --traceback

        >>> swap_solr_core('collection1', 'swap_core')
        >>> delete_solr_core('swap_core', delete_data_dir=True)

  - Bulk files have been rewritten and require some new tweaks:
        - Update cron to generate bulk files on the last day of each month,
          using something like the following:

              min hour 30 4,6,9,11        * manage.py cl_make_bulk_data
              min hour 31 1,3,5,7,8,10,12 * manage.py cl_make_bulk_data
              min hour 28 2               * manage.py cl_make_bulk_data
              min hour 30 4,6,9,11        * manage.py cl_make_citation_graph_bulk_data
              min hour 31 1,3,5,7,8,10,12 * manage.py cl_make_citation_graph_bulk_data
              min hour 28 2               * manage.py cl_make_citation_graph_bulk_data

        - Remove any old cron entries referencing dump_all_files.py
        - Add a new directory at $INSTALL_ROOT/alert/assets/media/bulk-data to
          contain the new bulk files when they are created, and symlink it from
          /sata/.
        - Generate the new bulk files by calling manage.py cl_make_bulk_data
          and manage.py cl_make_citation_graph_bulk_data.
        - Be sure that Apache is restarted so the new location is updated.
        - Delete any old bulk files located at
          $INSTALL_ROOT/alert/assets/media/dumps
  - Update cron jobs to point to cl_scrape_opinions and cl_scrape_oral_arguments
  - Update the CL calendar to have bulk files generated at 3am on the last day
    of every month.

2014-06-18:
 - This update makes alert editing a lot more intuitive.

2014-05-22:
 - This upgrade brings several major improvements including a new footer, social links, several new or updated pages, a
   new homepage and a feature tour. It's much-needed polish.
 - Several small apps have been combined into the "simple_pages" app. This should vastly simplify finding the right
   code for the task at hand but otherwise should not have any noticeable effects.
 - After upgrade, delete any flatpages that have been replaced by the new simple_pages app.

2013-11-25:
 - This revision adds a new script that will welcome new users whenever it is called. It will send an email to anybody
   that's new in the last 24 hours each time it is called. If you wish to use it, simply add it to your crontab.
 - This revision also tweaks Solr in a way that should not cause any problems for folks on 4.0 or 4.1, but that will
   make it perform faster in production.

2013-11-15:
 - This upgrades Solr to version 4.1, which supports file listeners. To upgrade Solr, simply:
   - Download Solr version 4.1 with:
        - cd ~/Downloads/
        - wget https://archive.apache.org/dist/lucene/solr/4.1.0/solr-4.1.0.tgz
   - Unpack the tgz file with:
        - tar -x -f solr-4.1.0.tgz
   - Set up the INSTALL_ROOT variable:
        - source /etc/courtlistener
   - Stop Solr, swap in the correct version and restart solr:
        - sudo service solr stop
        - sudo mv /usr/local/solr /usr/local/solr-4.0.bak && sudo mv ~/Downloads/solr-4.1.0 /usr/local/solr
        - sudo mv /usr/local/solr/example/solr/collection1/conf/solrconfig.xml /usr/local/solr/example/solr/collection1/conf/solrconfig.orig<commit_msg>Fixes to upgrade notes for OA.<commit_after>This file contains notes for upgrades when special instructions are necessary.
In general, simply doing:

# Get the latest code
git pull

# Gather any new static files
manage.py collectstatic (sudo -u www-data ./manage.py collectstatic)

(enter yes, when prompted)

# Migrate the database
manage.py migrate

# Restart celery
sudo service celeryd restart

# Restart apache's code
touch apache/django.wsgi

is all that's necessary. In the case that special instructions are needed, we
will attempt to put them here, with the most recent at the top.

We welcome a conversion of these notes to a better process using Fabric.

----------------------------


2014-10-23:
 Overview:
 - This is a very large upgrade and brings the concept of the "Docket" into
   CourtListener. After this point, we will have both opinions and oral
   arguments for the first time, with the result being that we need some kind
   of container that we can hold them in. A new class has been added to
   search.models called "Docket" to hold this kind of information.

 - When upgrading to this version, everything will completely break until your
   database is updated. I recommend enabling the maintenance mode URL if you
   wish to avoid chaos for your users.

 - The database upgrade for this version will take some time, as we are moving
   a number of fields from one database table to another. A re-index is also
   required, but, if you have ample disk space, this will not cause downtime.

  Deployment:
  - Install new dependencies:
    - sudo pip install --allow-external eyeD3 --allow-unverified eyeD3 eyeD3==0.7.4
    - sudo pip install datadiff==1.1.5
    - avconv: sudo apt-get install libav-tools
  - Install the seal-rookery:
    - sudo git clone https://github.com/freelawproject/seal-rookery /usr/local/seal_rookery
    - sudo ln -s /usr/local/seal_rookery /usr/lib/python2.7/dist-packages/seal_rookery

  - Disable any current scrapers using killall, and disable cron jobs from
    kicking in again.
  - Flush any background tasks with countdowns in the celery queue by following
    the notes here: https://stackoverflow.com/questions/26535719/
  - Stop Celery
  - Pull the code and the maintenance mode will be enabled.
  - Restart apache to enable the new code and clear its caches:
        - cl; sudo touch apache/django.wsgi
  - Database changes
        - manage.py migrate search --delete-ghost-migrations
        - manage.py migrate
  - Add a new core to Solr (paths below may need changing to correspond with
    the location of your current configurations):

        1. Create a new configuration directory based on our current one:

           % sudo cp -r /usr/local/solr/example/solr/collection1 /usr/local/solr/example/solr/audio
           % source /etc/courtlistener
           % sudo ln -s -f $INSTALL_ROOT/Solr/conf/audio_schema.xml /usr/local/solr/example/solr/audio/conf/schema.xml

        2. Create the new core in Python:

            from alert.lib.solr_core_admin import create_solr_core
            from alert import settings
            import os
            create_solr_core(
               core_name='audio',
               data_dir=os.path.join(settings.INSTALL_ROOT, 'Solr', 'data_audio'),
               schema=os.path.join(settings.INSTALL_ROOT, 'Solr', 'conf', 'audio_schema.xml'),
               instance_dir='/usr/local/solr/example/solr/audio',
            )

  - Turn maintenance mode back off.
  - Re-start celery
  - Re-enable scrapers and kick off the oral argument scrapers
  - Flip the has_oral_argument_scraper flag to true on all correct courts.
  - absolute_urls have changed throughout, necessitating a complete re-index
    for links in search results to work without redirects. We can re-index in
    a background core and then swap it in using the following commands:

        >>> create_solr_core(
            core_name='swap_core',
            data_dir=os.path.join(settings.INSTALL_ROOT, 'Solr', 'data_swap'),
            schema=os.path.join(settings.INSTALL_ROOT, 'Solr', 'conf', 'schema.xml'),
        )

        % screen
        % manage.py cl_update_index --update --everything --optimize --solr-url http://127.0.0.1:8983/solr/swap_core --type opinions --traceback

        >>> swap_solr_core('collection1', 'swap_core')
        >>> delete_solr_core('swap_core', delete_data_dir=True)

  - Bulk files have been rewritten and require some new tweaks:
        - Update cron to generate bulk files on the last day of each month,
          using something like the following:

              min hour 30 4,6,9,11        * manage.py cl_make_bulk_data
              min hour 31 1,3,5,7,8,10,12 * manage.py cl_make_bulk_data
              min hour 28 2               * manage.py cl_make_bulk_data

        - Remove any old cron entries referencing dump_all_files.py
        - Add a new directory at $INSTALL_ROOT/alert/assets/media/bulk-data to
          contain the new bulk files when they are created, and symlink it from
          /sata/.
        - Generate the new bulk files by calling manage.py cl_make_bulk_data.
        - Be sure that Apache is restarted so the new location is updated.
        - Delete any old bulk files located at
          $INSTALL_ROOT/alert/assets/media/dumps
  - Update cron jobs to point to cl_scrape_opinions and cl_scrape_oral_arguments
  - Update the CL calendar to have bulk files generated at 3am on the last day
    of every month.
  - Add symlinks for the new data types so we don't fill the SSDs

2014-06-18:
 - This update makes alert editing a lot more intuitive.

2014-05-22:
 - This upgrade brings several major improvements including a new footer, social links, several new or updated pages, a
   new homepage and a feature tour. It's much-needed polish.
 - Several small apps have been combined into the "simple_pages" app. This should vastly simplify finding the right
   code for the task at hand but otherwise should not have any noticeable effects.
 - After upgrade, delete any flatpages that have been replaced by the new simple_pages app.

2013-11-25:
 - This revision adds a new script that will welcome new users whenever it is called. It will send an email to anybody
   that's new in the last 24 hours each time it is called. If you wish to use it, simply add it to your crontab.
 - This revision also tweaks Solr in a way that should not cause any problems for folks on 4.0 or 4.1, but that will
   make it perform faster in production.

2013-11-15:
 - This upgrades Solr to version 4.1, which supports file listeners. To upgrade Solr, simply:
   - Download Solr version 4.1 with:
        - cd ~/Downloads/
        - wget https://archive.apache.org/dist/lucene/solr/4.1.0/solr-4.1.0.tgz
   - Unpack the tgz file with:
        - tar -x -f solr-4.1.0.tgz
   - Set up the INSTALL_ROOT variable:
        - source /etc/courtlistener
   - Stop Solr, swap in the correct version and restart solr:
        - sudo service solr stop
        - sudo mv /usr/local/solr /usr/local/solr-4.0.bak && sudo mv ~/Downloads/solr-4.1.0 /usr/local/solr
        - sudo mv /usr/local/solr/example/solr/collection1/conf/solrconfig.xml /usr/local/solr/example/solr/collection1/conf/solrconfig.orig
{'fraction_non_alphanumeric': 0.07317740391191356, 'fraction_numerical': 0.01798659554096567, 'mean_word_length': 2.737030411449016, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 21, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 10}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '593909', 'n_tokens_mistral': 4427, 'n_tokens_neox': 4124, 'n_words': 1705}