<commit_before>=================
PyCon 2010 script
=================

This is my "working script" -- I'll be using these commands, verbatim, as I
walk through the class. They probably only work on Ubuntu 9.10.

These notes are primarily for me only, so they may not make a whole lot of
sense without my narrative. However, for those not in the class, judicious use
of ``man`` and Google along with experimentation should make for an
interesting afternoon.

Pre-class
=========

Just to make sure everything's in place...

Boot the severs::

    local$ cloudservers boot oscon-web1 --key
    local$ cloudservers boot oscon-web2 --key
    local$ cloudservers boot oscon-db1 --key
    
``~/.ssh/config``::

    Host oscon-web1
        HostName XXX
        User root

    Host oscon-web2
        HostName XXX
        User root

    Host oscon-db1
        HostName XXX

Keys::

    local$ ssh-copy-id oscon-web1
    local$ ssh-copy-id oscon-web2
    local$ ssh-copy-id oscon-db1

Pre-fetch the big data file::

    web1$ wget XXXXXX

1. Getting started
==================

Initial setup::

    <all>$ aptitude update && aptitude -y safe-upgrade

Bootstrap up the example app::

    web1$ aptitude install git-core python-dev postgresql-client build-essential libpq-dev subversion
    web1$ mkdir -p /home/web/static
    web1$ cd /home/web
    web1$ git clone git://github.com/jacobian/django-deployment-workshop.git
    web1$ cd django-deployment-workshop/fumblerooski-site/
    web1$ python bootstrap.py
    web1$ ./bin/buildout
    web1$ ./bin/django runserver 0.0.0.0:8000
    
Get ``http://oscon-web1:8000/`` - operational error. Exactly what I'd expect.

2. PostgreSQL
=============
    
Install and configure Postgres::

    db1$ aptitude install git-core postgresql
    db1$ mkdir -p /home/web
    db1$ cd /home/web
    db1$ git clone git://github.com/jacobian/django-deployment-workshop.git
    db1$ cd /etc/postgresql/8.4/main/
    db1$ invoke-rc.d postgresql-8.4 stop
    db1$ mv postgresql.conf postgresql.conf.orig
    db1$ mv pg_hba.conf pg_hba.conf.orig
    db1$ ln -s /home/web/django-deployment-workshop/postgresql/postgresql.conf .
    db1$ ln -s /home/web/django-deployment-workshop/postgresql/pg_hba.conf .

Edit config files (``postrgesql.conf``, ``pg_hba.conf``) to include the
correct IP addresses, then start::

    db1$ vim pg_hba.conf
    db1$ vim postgresql.conf
    db1$ invoke-rc.d postgresql-8.4 start

And create users and database. Root user for convenience::

    db1$ sudo -u postgres createuser root
    db1$ createuser fumblerooski
    db1$ createdb -O fumblerooski fumblerooski

Wire up the app to PostgreSQL, load data, take a look::

    web1$ cd /home/web/django-deployment-workshop/fumblerooski-site
    web1$ vim fski/settings.py # DATABASE_HOST = '...'
    web1$ zcat /root/fumblerooski-dump.sql.gz | ./bin/django dbshell
    web1$ ./bin/django runserver 0.0.0.0:8000

Instead of installing the DB, can also ``./bin/django syncdb``.

Hit ``http://oscon-web1:8000/`` -- *boom*.

Take questions while the data loads.

3. mod_wsgi
===========

Install Apache and mod_wsgi, and zap away all the Ubuntu cruft::

    web1$ aptitude install apache2 libapache2-mod-wsgi
    web1$ cd /etc/apache2/
    web1$ rm -rf apache2.conf conf.d/ httpd.conf magic mods-* sites-* ports.conf 

Add our config. Remember to make a python eggs dir (it should be writable by
apache, but 777 is easier)::

    web1$ ln -s /home/web/django-deployment-workshop/apache/apache2.conf .
    web1$ mkdir -m777 -p /var/www/.python-eggs
    web1$ invoke-rc.d apache2 restart

Hit ``http://oscon-web1/`` -- *boom*.

4. fabfiles
===========

Run the fabfile::

    local$  ./bin/fab -f ../fabfiles/single-server.py deploy
    
5. Multiple web servers
=======================

Run the fabfile::

    local$  ./bin/fab -f ../fabfiles/multiple-webs.py setup:hosts=oscon-web2
    
While this is running, talk about configuration management.

Hit ``http://oscon-web2/`` -- *boom*.

6. Load balancers
=================

Set up Nginx on web1::

    web1$ aptitude install nginx 
    web1$ cd /etc/nginx/
    web1$ rm -rf conf.d/ fastcgi_params koi-* nginx.conf sites-* win-utf 
    web1$ ln -s /home/web/django-deployment-workshop/nginx/nginx.conf .

Edit Apache conf to listen on local IP, port 8000::

    web1$ vim /etc/apache2/apache2.conf
    web1$ invoke-rc.d apache2 restart

Edit Nginx to point to correct backend - just web1 for now::

    web1$ vim /etc/nginx/nginx.conf 
    web1$ invoke-rc.d nginx start

Check it out::

    local$ curl -I http://oscon-web1/

Edit Apache on web2 for local IP, port 8000::

    web2$ vim /etc/apache2/apache2.conf 
    web2$ invoke-rc.d apache2 restart

Edit Nginx to point to both backends::

    web1$ vim /etc/nginx/nginx.conf 
    web1$ invoke-rc.d nginx restart

Check it out, looking at ``X-Handled-By``::
    
    local$ curl -I http://oscon-web1/

7. Caching servers
==================

Install, enable, config memcached::

    web1$ aptitude install memcached
    web1$ echo "ENABLE_MEMCACHED=yes" > /etc/default/memcached
    web1$ rm /etc/memcached.conf
    web1$ ln -s /home/web/django-deployment-workshop/memcached/memcached.conf /etc/memcached.conf

Make memcached listen on the internal IP, then start it (and check)::

    web1$ vim /etc/memcached.conf 
    web1$ invoke-rc.d memcached start
    web1$ echo stats | nc -q1 <internal-ip> 11211

Repeat on web2.

Edit ``fski/settings.py`` to point to both cache servers::

    CACHE_BACKEND = "memcached://<web1>:11211;<web2>:11211/"
    
    MIDDLEWARE_CLASSES = (
        'django.middleware.cache.UpdateCacheMiddleware',
        ...
        'django.middleware.cache.FetchFromCacheMiddleware',
    )

Remember to do this on both servers.

Reload:

    web1$ invoke-rc.d apache2 reload
    web2$ invoke-rc.d apache2 reload

Check that it's working.

Check cache stats::

    web1$ echo stats | nc -q1 <web1> 11211
    web1$ echo stats | nc -q1 <web2> 11211

8. Database connection middleware
=================================

Install pgpool2. There's a bug in the current Ubuntu we'll need to work around
with a custom ``/etc/default/pgpool2``::

    web1$ aptitude install pgpool2
    web1$ pkill pgpool
    web1$ rm /etc/default/pgpool2
    web1$ ln -s /home/web/django-deployment-workshop/pgpool/etc-default-pgpool /etc/default/pgpool2
    web1$ rm /etc/pgpool.conf 
    web1$ ln -s /home/web/django-deployment-workshop/pgpool/pgpool-single-db.conf /etc/pgpool.conf

Fix up ``pgpool.conf`` to point to the backend (``db1``). Start it and check::

    web1$ vim /etc/pgpool.conf 
    web1$ invoke-rc.d pgpool2 start
    web1$ psql -l

Repeat on ``web2``.

Modify Django to use pgpool::

    DATABASE_HOST = ''

Again, remember both hosts. Reload:

    web1$ invoke-rc.d apache2 reload
    web2$ invoke-rc.d apache2 reload

Check it out.

Afterwards
==========

Stop the VMs::

    local$ cloudservers delete oscon-web1
    local$ cloudservers delete oscon-web2
    local$ cloudservers delete oscon-db1
    
Clean up ``/etc/hosts`` and ``~/.ssh/config``.<commit_msg>Oops, once last script fix.<commit_after>=================
PyCon 2010 script
=================

This is my "working script" -- I'll be using these commands, verbatim, as I
walk through the class. They probably only work on Ubuntu 9.10.

These notes are primarily for me only, so they may not make a whole lot of
sense without my narrative. However, for those not in the class, judicious use
of ``man`` and Google along with experimentation should make for an
interesting afternoon.

Pre-class
=========

Just to make sure everything's in place...

Boot the severs::

    local$ cloudservers boot oscon-web1 --key
    local$ cloudservers boot oscon-web2 --key
    local$ cloudservers boot oscon-db1 --key
    
``~/.ssh/config``::

    Host oscon-web1
        HostName XXX
        User root

    Host oscon-web2
        HostName XXX
        User root

    Host oscon-db1
        HostName XXX

Keys::

    local$ ssh-copy-id oscon-web1
    local$ ssh-copy-id oscon-web2
    local$ ssh-copy-id oscon-db1

Pre-fetch the big data file::

    web1$ wget XXXXXX

1. Getting started
==================

Initial setup::

    <all>$ aptitude update && aptitude -y safe-upgrade

Bootstrap up the example app::

    web1$ aptitude install git-core python-dev postgresql-client build-essential libpq-dev subversion
    web1$ mkdir -p /home/web/static
    web1$ cd /home/web
    web1$ git clone git://github.com/jacobian/django-deployment-workshop.git
    web1$ cd django-deployment-workshop/fumblerooski-site/
    web1$ python bootstrap.py
    web1$ ./bin/buildout
    web1$ ./bin/django runserver 0.0.0.0:8000
    
Get ``http://oscon-web1:8000/`` - operational error. Exactly what I'd expect.

2. PostgreSQL
=============
    
Install and configure Postgres::

    db1$ aptitude install git-core postgresql
    db1$ mkdir -p /home/web
    db1$ cd /home/web
    db1$ git clone git://github.com/jacobian/django-deployment-workshop.git
    db1$ cd /etc/postgresql/8.4/main/
    db1$ invoke-rc.d postgresql-8.4 stop
    db1$ mv postgresql.conf postgresql.conf.orig
    db1$ mv pg_hba.conf pg_hba.conf.orig
    db1$ ln -s /home/web/django-deployment-workshop/postgresql/postgresql.conf .
    db1$ ln -s /home/web/django-deployment-workshop/postgresql/pg_hba.conf .

Edit config files (``postrgesql.conf``, ``pg_hba.conf``) to include the
correct IP addresses, then start::

    db1$ vim pg_hba.conf
    db1$ vim postgresql.conf
    db1$ invoke-rc.d postgresql-8.4 start

And create users and database. Root user for convenience::

    db1$ sudo -u postgres createuser root
    db1$ createuser fumblerooski
    db1$ createdb -O fumblerooski fumblerooski

Wire up the app to PostgreSQL, load data, take a look::

    web1$ cd /home/web/django-deployment-workshop/fumblerooski-site
    web1$ vim fski/settings.py # DATABASE_HOST = '...'
    web1$ zcat /root/fumblerooski-dump.sql.gz | ./bin/django dbshell
    web1$ ./bin/django runserver 0.0.0.0:8000

Instead of installing the DB, can also ``./bin/django syncdb``.

Hit ``http://oscon-web1:8000/`` -- *boom*.

Take questions while the data loads.

3. mod_wsgi
===========

Install Apache and mod_wsgi, and zap away all the Ubuntu cruft::

    web1$ aptitude install apache2 libapache2-mod-wsgi
    web1$ cd /etc/apache2/
    web1$ rm -rf apache2.conf conf.d/ httpd.conf magic mods-* sites-* ports.conf 

Add our config. Remember to make a python eggs dir (it should be writable by
apache, but 777 is easier)::

    web1$ ln -s /home/web/django-deployment-workshop/apache/apache2.conf .
    web1$ mkdir -m777 -p /var/www/.python-eggs
    web1$ invoke-rc.d apache2 restart

Hit ``http://oscon-web1/`` -- *boom*.

4. fabfiles
===========

Run the fabfile::

    local$  fab -f fabfiles/single-server.py deploy
    
5. Multiple web servers
=======================

Run the fabfile::

    local$  fab -f fabfiles/multiple-webs.py setup:hosts=oscon-web2
    
While this is running, talk about configuration management.

Hit ``http://oscon-web2/`` -- *boom*.

6. Load balancers
=================

Set up Nginx on web1::

    web1$ aptitude install nginx 
    web1$ cd /etc/nginx/
    web1$ rm -rf conf.d/ fastcgi_params koi-* nginx.conf sites-* win-utf 
    web1$ ln -s /home/web/django-deployment-workshop/nginx/nginx.conf .

Edit Apache conf to listen on local IP, port 8000::

    web1$ vim /etc/apache2/apache2.conf
    web1$ invoke-rc.d apache2 restart

Edit Nginx to point to correct backend - just web1 for now::

    web1$ vim /etc/nginx/nginx.conf 
    web1$ invoke-rc.d nginx start

Check it out::

    local$ curl -I http://oscon-web1/

Edit Apache on web2 for local IP, port 8000::

    web2$ vim /etc/apache2/apache2.conf 
    web2$ invoke-rc.d apache2 restart

Edit Nginx to point to both backends::

    web1$ vim /etc/nginx/nginx.conf 
    web1$ invoke-rc.d nginx restart

Check it out, looking at ``X-Handled-By``::
    
    local$ curl -I http://oscon-web1/

7. Caching servers
==================

Install, enable, config memcached::

    web1$ aptitude install memcached
    web1$ rm /etc/memcached.conf
    web1$ ln -s /home/web/django-deployment-workshop/memcached/memcached.conf /etc/memcached.conf

Make memcached listen on the internal IP, then start it (and check)::

    web1$ vim /etc/memcached.conf 
    web1$ invoke-rc.d memcached restart
    web1$ echo stats | nc -q1 <internal-ip> 11211

Repeat on web2.

Edit ``fski/settings.py`` to point to both cache servers::

    CACHE_BACKEND = "memcached://<web1>:11211;<web2>:11211/"
    
    MIDDLEWARE_CLASSES = (
        'django.middleware.cache.UpdateCacheMiddleware',
        ...
        'django.middleware.cache.FetchFromCacheMiddleware',
    )

Remember to do this on both servers.

Reload:

    web1$ invoke-rc.d apache2 reload
    web2$ invoke-rc.d apache2 reload

Check that it's working.

Check cache stats::

    web1$ echo stats | nc -q1 <web1> 11211
    web1$ echo stats | nc -q1 <web2> 11211

8. Database connection middleware
=================================

Install pgpool2. There's a bug in the current Ubuntu we'll need to work around
with a custom ``/etc/default/pgpool2``::

    web1$ aptitude install pgpool2
    web1$ pkill pgpool
    web1$ rm /etc/default/pgpool2
    web1$ ln -s /home/web/django-deployment-workshop/pgpool/etc-default-pgpool /etc/default/pgpool2
    web1$ rm /etc/pgpool.conf 
    web1$ ln -s /home/web/django-deployment-workshop/pgpool/pgpool-single-db.conf /etc/pgpool.conf

Fix up ``pgpool.conf`` to point to the backend (``db1``). Start it and check::

    web1$ vim /etc/pgpool.conf 
    web1$ invoke-rc.d pgpool2 start
    web1$ psql -l

Repeat on ``web2``.

Modify Django to use pgpool::

    DATABASE_HOST = ''

Again, remember both hosts. Reload:

    web1$ invoke-rc.d apache2 reload
    web2$ invoke-rc.d apache2 reload

Check it out.

Afterwards
==========

Stop the VMs::

    local$ cloudservers delete oscon-web1
    local$ cloudservers delete oscon-web2
    local$ cloudservers delete oscon-db1
    
Clean up ``/etc/hosts`` and ``~/.ssh/config``.
{'fraction_non_alphanumeric': 0.13218676710965815, 'fraction_numerical': 0.029351147750692915, 'mean_word_length': 3.0091168091168092, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1994050', 'n_tokens_mistral': 5151, 'n_tokens_neox': 4799, 'n_words': 1497}