<commit_before>Setting up the database
=======================

1. Set up a PostgreSQL database with the PostGIS extension (see [PostGIS setup](http://wiki.openstreetmap.org/wiki/Osmosis/PostGIS_Setup) document for help with that).
2. Run the `sql/pgsnapshot_schema_0.6.sql` script in that database.
3. Run the `sql/pgsnapshot_schema_0.6_linestring.sql` script in that database.
4. Run the `sql/owl_schema.sql` script in that database.

Initial data import
===================

If you don't want to process all OSM data changes "since forever" that will build up your database from scratch, you can start from any point in time. In order to do that, you need to do an initial data import.

OWL database schema is compatible with the Osmosis pgsnapshot schema so you can/should use the `--write-pgsql` [Osmosis task](http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--write-pgsql_.28--wp.29) to import data.

Installing the OWL Osmosis plugin
=================================

OWL plugin for Osmosis is needed for the `--write-owldb-change` task which populates OWL tables with data.

To install the plugin just drop the JAR file from the `osmosis-plugin` directory to the `lib` directory in your Osmosis installation (e.g. `/usr/share/osmosis`).

If you want, you can also build the plugin from the source]():

1. Clone [the source code](https://github.com/ppawel/osmosis/tree/owldb-plugin).
2. Type `ant build`.
3. In the `package` directory you should now have a full Osmosis distribution ready to be used (the plugin itself is in the `package/lib/default` directory).

Populating the database with changes
====================================

In order to start populating OWL tables, you need to process OsmChange files against your database.

The easiest way to do this is to set up the Osmosis interval replication.

First, [initialize replication directory](http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--read-replication-interval-init_.28--rrii.29).

Then, set up (e.g. in crontab) the replication pipeline:

    osmosis --read-replication-interval workingDirectory=<dir> --tee-change  --write-owldb-change authFile=<authFile> --write-pgsql-change authFile=<authFile>

This commands does couple of things:
* Downloads OsmChange file for a specific replication interval (minute/hour/day) - according to the configuration in the `configuration.txt` file.
* The change stream goes to the OWL Osmosis plugin which populates OWL tables and then applies changes to regular data tables (same as with the `--write-pgsql-change` task).<commit_msg>Update installation instructions after running through them.<commit_after>Setting up the database
=======================

Installation instructions for PostgreSQL 9.1, for equivalent 8.x procedures
refer to http://wiki.openstreetmap.org/wiki/Osmosis/PostGIS_Setup.

1) Set up PostGIS database

    # Assuming a database owl here, can be any other database name.
    postgres createdb owl
    postgres createlang plpgsql owl
    postgres psql -f /path/to/pgsql/share/contrib/postgis-2.0/postgis.sql owl
    postgres psql -f /path/to/pgsql/share/contrib/postgis-2.0/spatial_ref_sys.sql owl

2) Create hstore extension

    postgres psql -c "CREATE EXTENSION hstore;" owl

3) Set up pg snapshot schema

    postgres psql -f sql/pgsnapshot_schema_0.6.sql
    postgres psql -f sql/pgsnapshot_schema_0.6_linestring.sql

4) Install OWL schema

    postgres psql -f sql/owl_schema.sql

Install Osmosis
===============

For now we're using a modified version of Osmosis, this should just be an Osmosis plugin in the future.

    git clone -b owldb-plugin https://github.com/ppawel/osmosis.git
    ant build
    # Copy resulting build to a path of your convenience

Initial data import
===================

*From Planet file*

You can save time by not doing a full history import and use the lighter Planet file without historic changesets from [planet.osm.org](http://planet.osm.org/).

The OWL database schema is compatible with the Osmosis pgsnapshot schema so you can use the [Osmosis `--write-pgsql` task](http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--write-pgsql_.28--wp.29) to import OSM data.

    # Assuming a database owl and a user postgres
    curl -o planet.osm.bz2 http://planet.osm.org/planet/planet-latest.osm.bz2
    bunzip2 planet.osm.bz2
    osmosis --read-xml planet.osm --write-pgsql database=owl user=postgres

*From full history Planet file*

TODO: Is this supported at all right now?

Populating the database with changes
====================================

In order to start populating OWL tables, you need to process OsmChange files against your database.

The easiest way to do this is to set up the Osmosis interval replication.

1) [Initialize replication directory](http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--read-replication-interval-init_.28--rrii.29).

2) Set up (e.g. in crontab) the replication pipeline:

    # Make sure to use the OWL specific osmosis version built above
    osmosis \
    --read-replication-interval workingDirectory=replication/ \
    --tee-change \
    --write-owldb-change database=owl user=postgres \
    --write-pgsql-change database=owl user=postgres

This command:

- downloads OsmChange file for a specific replication interval (minute/hour/day) - according to the configuration in the `configuration.txt` file.
- populates OWL tables and then applies changes to regular data tables (same as `--write-pgsql-change` task).
{'fraction_non_alphanumeric': 0.11575680116852291, 'fraction_numerical': 0.008763921855030126, 'mean_word_length': 4.222116301239275, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '849947', 'n_tokens_mistral': 1684, 'n_tokens_neox': 1618, 'n_words': 618}