<commit_before>*****************
django-deb-deploy
*****************

**this is a work in progress, it is not yet completed. A working version
should be released in the next week or two.**

This is a tool to create .deb packages for deploying Django sites on
production Debian systems.

For now, the following is assumed/configured:

* A debian package for the virtualenv, created from the requirements.txt file
* A debian package for the version of your source code
* A debian package for your site, which depends on the above packages and contains:

   - nginx configuration
   - systemd configuration for the site's process (gunicorn) and any celery
     workers
   - postinst script to ensure database, users etc. exist, to switch
     running gunicorn process to new software version atomically, and to
     optionally run migrate (default off!)

A new package name is created for each possible virtualenv configuration and
source code version, instead of incrementing the version number of a single
package. This means that the production system can host multiple versions
at the same time, which is good for multi-tenancy and seamless migration from
one version to the next without interruption.

Out of date packages can be eventually removed automatically using debian
tools (i.e. apt-get autoremove).


Configuration
=============

Your code needs to define the following:

* requirements.txt
* venv-debian-requirements.txt (required debian packages for virtualenv packages)
* src-debian-requirements.txt (required debian packages for src)
* When running the script, you will need to provide arguments to communicate:

   - the software's name (used in package and script names, keep it short, no spaces etc)
   - the software's variant (optional, for multitenancy)
   - the version hash to be built
   - the build directory (see below for creating the build directory)

To create a build directory run eg ``sudo django-deb-deploy create --dir /path/to/build-dir/``. This will:

* create a debbootstrap instance in the given directory
* create the required system users and groups
* install schroot configuration to allow normal users to use the bootstrapped debian instance
* install any extra required debian packages for building

Then run ``django-deb-deploy init mysoftware --dir /path/to/build-dir/ --clone git+http://server.com/git/repository``. This will:

* clone your source code repository

Not that the build machine must be the same architecture and use the same debian variant (``jessie`` in the example) as the target system.

To create the required packages run eg ``django-deb-deploy build mysoftware --dir /path/to/build-dir --variant berlin --version 1a2b3c4d --settings path-to-settings-module``. This will:

* Install any required debian packages [1]
* Run ``git fetch`` on the repository
* Checkout the requested version into its place
* Build the requirements package if necessary (or wait for it to be built)
* Build the software version package if necessary (or wait for it to be built)
* Run ``./manage.py collectstatic`` (saved into the site package's directory)
* Build the (variant's) site package

.. [1] If no other builds are running there will be no issues, but if another build is running and upgrades or package uninstalls are required, then it will wait for the other package build to finish before starting.


virtualenv package
==================
A package is generated that contains a virtualenv that is required for this version.

* a hash is generated from the ``requirements.txt`` file, after it is sorted and de-commented
* this hash will appear in the name of the debian package, e.g. ``{software name}-env-{hash}``
* the virtualenv is created and saved to /usr/lib/{software name}-env/{hash}/ in the build directory
* the requirements.txt file is saved to /usr/share/{software name}-env/{hash}/requirements.txt

This package will contain the (binary) files already in place for the debian machine. It will probably be large, but will not need to be installed for every upgrade, only the upgrades where the ``requirements.txt`` file has substantively changed. Because the python libraries will be compiled, you must build on the same machine type and debian install as the target system.


Source package
==============
The source is installed separately to the virtualenv because it is updated more often that the virtualenv package. This helps keep updates smaller, but means that the source needs to be included in the PYTHONPATH for it to be accessible.<commit_msg>More readme tweaks<commit_after>*****************
django-deb-deploy
*****************

**this is a work in progress, it is not yet completed. A working version
should be released in the next week or two.**

This is a tool to create .deb packages for deploying Django sites on
production Debian systems.

For now, the following is assumed/configured:

* A debian package for the virtualenv, created from the requirements.txt file
* A debian package for the version of your source code
* A debian package for your site, which depends on the above packages and contains:

  - nginx configuration
  - systemd configuration for the site's process (gunicorn) and any celery
    workers
  - postinst script to ensure database, users etc. exist, to switch
    running gunicorn process to new software version atomically, and to
    optionally run migrate (default off!)

A new package name is created for each possible virtualenv configuration and
source code version, instead of incrementing the version number of a single
package. This means that the production system can host multiple versions
at the same time, which is good for multi-tenancy and seamless migration from
one version to the next without interruption.

Out of date packages can be eventually removed automatically using debian
tools (i.e. apt-get autoremove).


Configuration
=============

Your code needs to define the following:

* requirements.txt
* venv-debian-requirements.txt (required debian packages for virtualenv packages)
* src-debian-requirements.txt (required debian packages for src)
* When running the script, you will need to provide arguments to communicate:

  - the software's name (used in package and script names, keep it short, no spaces etc)
  - the software's variant (optional, for multitenancy)
  - the version hash to be built
  - the build directory (see below for creating the build directory)

To create a build directory run eg ``sudo django-deb-deploy create --dir /path/to/build-dir/``. This will:

* create a debbootstrap instance in the given directory
* create the required system users and groups
* install schroot configuration to allow normal users to use the bootstrapped debian instance
* install any extra required debian packages for building

Then run ``django-deb-deploy init mysoftware --dir /path/to/build-dir/ --clone git+http://server.com/git/repository``. This will:

* clone your source code repository

Not that the build machine must be the same architecture and use the same debian variant (``jessie`` in the example) as the target system.

To create the required packages run eg ``django-deb-deploy build mysoftware --dir /path/to/build-dir --variant berlin --version 1a2b3c4d --settings path-to-settings-module``. This will:

* Install any required debian packages [1]_
* Run ``git fetch`` on the repository
* Checkout the requested version into its place
* Build the requirements package if necessary (or wait for it to be built)
* Build the software version package if necessary (or wait for it to be built)
* Run ``./manage.py collectstatic`` (saved into the site package's directory)
* Build the (variant's) site package

.. [1] If no other builds are running there will be no issues, but if another build is running and upgrades or package uninstalls are required, then it will wait for the other package build to finish before starting.


virtualenv package
==================
A package is generated that contains a virtualenv that is required for this version.

* a hash is generated from the ``requirements.txt`` file, after it is sorted and de-commented
* this hash will appear in the name of the debian package, e.g. ``{software name}-env-{hash}``
* the virtualenv is created and saved to /usr/lib/{software name}-env/{hash}/ in the build directory
* the requirements.txt file is saved to /usr/share/{software name}-env/{hash}/requirements.txt

This package will contain the (binary) files already in place for the debian machine. It will probably be large, but will not need to be installed for every upgrade, only the upgrades where the ``requirements.txt`` file has substantively changed. Because the python libraries will be compiled, you must build on the same machine type and debian install as the target system.


Source package
==============
The source is installed separately to the virtualenv because it is updated more often that the virtualenv package. This helps keep updates smaller, but means that the source needs to be included in the PYTHONPATH for it to be accessible.
{'fraction_non_alphanumeric': 0.0738774607387746, 'fraction_numerical': 0.0013271400132714001, 'mean_word_length': 4.392367322599881, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1712298', 'n_tokens_mistral': 2247, 'n_tokens_neox': 2125, 'n_words': 1275}