<commit_before>DIRTSAND - The D'ni In Real-Time Server And Network Dæmon
=========================================================

Introduction
------------

DIRTSAND is a full featured MOULa-compatible server platform for POSIX-compliant
operating systems, written in C++ and released under the AGPL version 3+.
Currently, it has only been tested on Linux, but in theory it should work on
other Unixes as well.  There are, however, currently no plans for Windows
development or support.


Prerequisites
-------------
* GCC 4.7+ (might work with other C++11 compliant compilers, but untested)
* Postgres (libpq) -- the actual database server can reside anywhere
* OpenSSL
* libreadline
* zlib
* [string_theory](https://github.com/zrax/string_theory)
* git (to get the sources)


Building the code
-----------------
1) Create a user and path for the DIRTSAND server to run from:

   ```
   $ sudo useradd dirtsand -d /opt/dirtsand -m -p <password> -s /bin/bash
   $ su dirtsand
   ```

2) Check out a copy of the source:

   ```
   $ git clone git://github.com/H-uru/dirtsand.git dirtsand
   $ cd dirtsand
   ```

3) Compile it for your system:

   ```
   $ mkdir build && cd build
   $ cmake -DCMAKE_INSTALL_PREFIX=/opt/dirtsand ..
   $ make && sudo make install
   $ cd ..
   ```

   If you run into any errors about finding libraries or headers, make sure
   you have the *development* versions of all of the required libraries, and
   that they are in your path.  You can also use the `cmake-gui` to help cmake
   locate the missing paths and files.


Setting up a server
-------------------

You will need a working Postgres server which DIRTSAND can use to store its
data.  Although you don't need superuser access to the postgres server, you
will need to have a database which you can add schemas, tables, etc into.

Setting up postgres functionality without a superuser account is currently
beyond the scope of this document...  If you need to do this, keep in mind
that you may also have to edit the database initialization scripts to work
against an existing user and/or database.

For the default installation, the provided scripts will create a dirtsand
database and set its ownership to a 'dirtsand' database user, which can
directly map the system dirtsand user created in step 1 of the "Building
the code" instructions above.  For better security, it is recommended to
use a password (as shown in the steps below), which can be configured in
the server settings as described in the "configure dirtsand" step.


1) Set up the postgres user:

   ```
   $ sudo -u postgres psql -d template1
   template1=# CREATE USER dirtsand WITH PASSWORD '<password>';
   template1=# CREATE DATABASE dirtsand WITH TEMPLATE = template0 ENCODING = 'UTF8';
   template1=# ALTER DATABASE dirtsand OWNER TO dirtsand;
   template1=# \q
   ```

2) Install the UUID functionality:

   This may be provided by your OS distribution.  In Ubuntu, simply install
   the `postgresql-contrib` package to provide the necessary libraries and
   installation scripts.  If your distribution does not provide a contrib
   or uuid-ossp bundle, you can get it and build it yourself from the
   sources provided at:  http://www.ossp.org/pkg/lib/uuid/

   In versions of Postgres **older than** 9.x, you will need to use the
   import script to add the uuid extension to the dirtsand database:

   ```
   $ sudo -u postgres psql -d dirtsand < /path/to/uuid-ossp.sql
   
   # (Example for Ubuntu 10.10 with PostgreSQL 8.4)
   $ sudo -u postgres psql -d dirtsand < /usr/share/postgresql/8.4/contrib/uuid-ossp.sql
   ```

   In PostgreSQL 9.x and newer, simply use the "CREATE EXTENSION" syntax to
   add the uuid-ossp extension it to the Dirtsand database:

   ```
   $ sudo -u postgres psql -d dirtsand
   dirtsand=# CREATE EXTENSION "uuid-ossp";
   dirtsand=# \q
   ```

3) Set up the dirtsand database:

   ```
   $ psql -d dirtsand < db/dbinit.sql
   $ psql -d dirtsand < db/functions.sql
   ```

   If there were no errors, your database should be ready for DIRTSAND.

4) Configure dirtsand:

   A sample dirtsand.ini has been provided in the root of the dirtsand
   sources.  We can copy this to our install directory and then edit the
   fields we need.  Specifically, you will need to adjust the server
   addresses and the RC4 keys.  If you have dirtsand installed to somewhere
   other than /opt/dirtsand, you will also need to point the configuration
   to the right paths too.

    ```
   $ sudo cp dirtsand.sample.ini /opt/dirtsand/dirtsand.ini
   $ sudo chown dirtsand /opt/dirtsand/dirtsand.ini
   $ <your-favorite-editor> dirtsand.ini
   ```

   To generate the RC4 keys, you can simply run the keygen command from
   within the dirtsand interactive console:

   ```
   $ bin/dirtsand
   ds-902> keygen new       (This one will take a little while)
   ds-902> quit
   ```

   Any errors that dirtsand spits out about config files and postgres
   passwords can be ignored, since you haven't provided a configuration
   file yet.

   You should now have a bunch of keys output on your terminal.  The first
   block (labeled Server keys) is the set you should paste into your
   dirtsand.ini.  Replace the dummy lines (with the '...' values) with the
   output from the `keygen new` command.  The second set of keys can be
   placed directly in the client's server.ini file (NOTE: This requires
   either the H-uru fork of CWE or PlasmaClient -- the vanilla MOUL
   client cannot load keys from a file, and you will have to enter the
   keys as byte arrays directly into the executable.

5) Provide data for the client to use:<commit_msg>Update README to simplify key generation.<commit_after>DIRTSAND - The D'ni In Real-Time Server And Network Dæmon
=========================================================

Introduction
------------

DIRTSAND is a full featured MOULa-compatible server platform for POSIX-compliant
operating systems, written in C++ and released under the AGPL version 3+.
Currently, it has only been tested on Linux, but in theory it should work on
other Unixes as well.  There are, however, currently no plans for Windows
development or support.


Prerequisites
-------------
* GCC 4.7+ (might work with other C++11 compliant compilers, but untested)
* Postgres (libpq) -- the actual database server can reside anywhere
* OpenSSL
* libreadline
* zlib
* [string_theory](https://github.com/zrax/string_theory)
* git (to get the sources)


Building the code
-----------------
1) Create a user and path for the DIRTSAND server to run from:

   ```
   $ sudo useradd dirtsand -d /opt/dirtsand -m -p <password> -s /bin/bash
   $ su dirtsand
   ```

2) Check out a copy of the source:

   ```
   $ git clone git://github.com/H-uru/dirtsand.git dirtsand
   $ cd dirtsand
   ```

3) Compile it for your system:

   ```
   $ mkdir build && cd build
   $ cmake -DCMAKE_INSTALL_PREFIX=/opt/dirtsand ..
   $ make && sudo make install
   $ cd ..
   ```

   If you run into any errors about finding libraries or headers, make sure
   you have the *development* versions of all of the required libraries, and
   that they are in your path.  You can also use the `cmake-gui` to help cmake
   locate the missing paths and files.


Setting up a server
-------------------

You will need a working Postgres server which DIRTSAND can use to store its
data.  Although you don't need superuser access to the postgres server, you
will need to have a database which you can add schemas, tables, etc into.

Setting up postgres functionality without a superuser account is currently
beyond the scope of this document...  If you need to do this, keep in mind
that you may also have to edit the database initialization scripts to work
against an existing user and/or database.

For the default installation, the provided scripts will create a dirtsand
database and set its ownership to a 'dirtsand' database user, which can
directly map the system dirtsand user created in step 1 of the "Building
the code" instructions above.  For better security, it is recommended to
use a password (as shown in the steps below), which can be configured in
the server settings as described in the "configure dirtsand" step.


1) Set up the postgres user:

   ```
   $ sudo -u postgres psql -d template1
   template1=# CREATE USER dirtsand WITH PASSWORD '<password>';
   template1=# CREATE DATABASE dirtsand WITH TEMPLATE = template0 ENCODING = 'UTF8';
   template1=# ALTER DATABASE dirtsand OWNER TO dirtsand;
   template1=# \q
   ```

2) Install the UUID functionality:

   This may be provided by your OS distribution.  In Ubuntu, simply install
   the `postgresql-contrib` package to provide the necessary libraries and
   installation scripts.  If your distribution does not provide a contrib
   or uuid-ossp bundle, you can get it and build it yourself from the
   sources provided at:  http://www.ossp.org/pkg/lib/uuid/

   In versions of Postgres **older than** 9.x, you will need to use the
   import script to add the uuid extension to the dirtsand database:

   ```
   $ sudo -u postgres psql -d dirtsand < /path/to/uuid-ossp.sql

   # (Example for Ubuntu 10.10 with PostgreSQL 8.4)
   $ sudo -u postgres psql -d dirtsand < /usr/share/postgresql/8.4/contrib/uuid-ossp.sql
   ```

   In PostgreSQL 9.x and newer, simply use the "CREATE EXTENSION" syntax to
   add the uuid-ossp extension it to the Dirtsand database:

   ```
   $ sudo -u postgres psql -d dirtsand
   dirtsand=# CREATE EXTENSION "uuid-ossp";
   dirtsand=# \q
   ```

3) Set up the dirtsand database:

   ```
   $ psql -d dirtsand < db/dbinit.sql
   $ psql -d dirtsand < db/functions.sql
   ```

   If there were no errors, your database should be ready for DIRTSAND.

4) Configure dirtsand:

   A sample dirtsand.ini has been provided in the root of the dirtsand
   sources.  We can copy this to our install directory and then edit the
   fields we need.  Specifically, you will need to adjust the server
   addresses and the RC4 keys.  If you have dirtsand installed to somewhere
   other than /opt/dirtsand, you will also need to point the configuration
   to the right paths too.

    ```
   $ sudo cp dirtsand.sample.ini /opt/dirtsand/dirtsand.ini
   $ sudo chown dirtsand /opt/dirtsand/dirtsand.ini
   $ <your-favorite-editor> dirtsand.ini
   ```

   To generate the RC4 keys, you can simply run the command-line option
   `generate-keys` using dirtsand:

   ```
   $ bin/dirtsand --generate-keys
   ```

   You should now have a bunch of keys output on your terminal.  The first
   block (labeled Server keys) is the set you should paste into your
   dirtsand.ini.  Replace the dummy lines (with the '...' values) with the
   output from the `generate-keys` command.  The second set of keys can be
   placed directly in the client's server.ini file (NOTE: This requires
   either the H-uru fork of CWE or PlasmaClient -- the vanilla MOUL
   client cannot load keys from a file, and you will have to enter the
   keys as byte arrays directly into the executable.

5) Provide data for the client to use:
{'fraction_non_alphanumeric': 0.08673607967406066, 'fraction_numerical': 0.006518786781349027, 'mean_word_length': 3.224091778202677, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7161375', 'n_tokens_mistral': 3336, 'n_tokens_neox': 3147, 'n_words': 1530}