<commit_before># Hecuba

A data platform built with [Clojure][CLJ], [ClojureScript][CLJS], [core.async][CORE.ASYNC], [Om][OM], Cassandra and other technologies.

## Build Status

[![Build Status](https://travis-ci.org/MastodonC/kixi.hecuba.png?branch=master)](https://travis-ci.org/MastodonC/kixi.hecuba)

## Usage

To start the server, clone the repo, and then lein repl or cider-jack-in as you prefer

```
(go)
```

When you make changes just reset.

```
(reset)
```

Then you'll need to compile the clojurescipt. That can be done with
cljsbuild like this

```
lein cljsbuild auto hecuba
```

This will recompile your clojurescript each time you save. The server
app needs to be bounced by doing ```reset``` as above.

If you pull from github, you'll still probably want to restart your
nrepl session though.

To clean the build do:

```
lein clean
```

This will delete both clj and cljs from `target` and `out` directories.

You can also use figwheel:
Instead of running ``` lein cljsbuild auto hecuba``` and reloading the
website, update ``` project.clj ``` to have this:

```
:env {:is-dev true}

```
and run:

```
lein figwheel hecuba
```

Each time you save your cljs files, figwheel will recompile and
refresh the website for you.

Environment should be set to ``` {:is-dev false} ``` before
deployment to the server.
For production deployment please compile once:

```
lein cljsbuild once hecuba
```

To run Clojure tests do:

```
lein test
```

To run ClojureScript tests:

```
lein cljsbuild test
```

## Dev environment

We are using Vagrant to manage dev environments.

+ Install VirtualBox v4.3.28 (correct as of 10th June 2015) from [here](https://www.virtualbox.org/wiki/Downloads) or preferably via your OS's package manager.
+ Install Vagrant v. 1.7.2 (correct as of 10th June 2015) from [here](http://www.vagrantup.com/) or preferably via your OS's package manager.
+ install the vbguest plugin so Virtual Box guest additions will updated
  for you ``vagrant plugin install vagrant-vbguest``
+ ``cd ${PROJECT_HOME}``
+ ``vagrant up`` (This will download stuff the first time and will be slow, after that it will be quicker)
+ You will now have all the services required running in a virtual machine with the ports forwarded for access from your local machine
+ You will need to install a cassandra _client_ only on your local machine. (You should not be running a cassandra locally as the ports will clash with those forwarded from the vagrant machine).

### First Time Test Data

+ Work out what the hostname you should use to connect to cassandra is. Look at the output of ``netstat -tln``, find the line that says something like ``aaa.bbb.ccc.ddd:9042``. (note this might be an ipv6 address depending on how your network is configured). This is the address to use in your hecuba config file and in the commands below.
+ create the test schema ``cqlsh aaa.bbb.ccc.ddd -f hecuba-schema.sql`` (this might show an error the first time you run it about the test namespace not existing, you can ignore that error)
+ Start cqlsh with the right hostname ``cqlsh aaa.bbb.ccc.ddd``
+ Create the Elastic Search schema by running ``scripts/build-es-schema.sh``
+ Create a file ``~/.hecuba.edn``` with the following contents:
```
{
 :cassandra-session {:keyspace :test}
 :hecuba-session {:keyspace :test}
 :search-session {:host :host "<the address you found above>" :name "hecuba"}
 :s3          {:access-key "<your personal AWS access key DO NOT SHARE KEYS!>"
               :secret-key "<your personal AWS secret key DO NOT SHARE KEYS!>"
               ;; you will need to create these buckets.
               :file-bucket "mc-<yourname>-hecuba-uploads"
               :status-bucket "mc-<yourname>-hecuba-status"
               :download-dir "/tmp"}
               
}
```

Back on your host machine do the following:

+ Start a repl in your favourite way.
+ Start the application with (go)
+ (require 'etl)
+ (kixi.hecuba.security/add-user! (:store kixipipe.application/system) "Mastodon" "<EMAIL>" "password" :kixi.hecuba.security/super-admin  #{} #{} )<commit_msg>Add clarification about where things run.<commit_after># Hecuba

A data platform built with [Clojure][CLJ], [ClojureScript][CLJS], [core.async][CORE.ASYNC], [Om][OM], Cassandra and other technologies.

## Build Status

[![Build Status](https://travis-ci.org/MastodonC/kixi.hecuba.png?branch=master)](https://travis-ci.org/MastodonC/kixi.hecuba)

## Usage

To start the server, clone the repo, and then lein repl or cider-jack-in as you prefer

```
(go)
```

When you make changes just reset.

```
(reset)
```

Then you'll need to compile the clojurescipt. That can be done with
cljsbuild like this

```
lein cljsbuild auto hecuba
```

This will recompile your clojurescript each time you save. The server
app needs to be bounced by doing ```reset``` as above.

If you pull from github, you'll still probably want to restart your
nrepl session though.

To clean the build do:

```
lein clean
```

This will delete both clj and cljs from `target` and `out` directories.

You can also use figwheel:
Instead of running ``` lein cljsbuild auto hecuba``` and reloading the
website, update ``` project.clj ``` to have this:

```
:env {:is-dev true}

```
and run:

```
lein figwheel hecuba
```

Each time you save your cljs files, figwheel will recompile and
refresh the website for you.

Environment should be set to ``` {:is-dev false} ``` before
deployment to the server.
For production deployment please compile once:

```
lein cljsbuild once hecuba
```

To run Clojure tests do:

```
lein test
```

To run ClojureScript tests:

```
lein cljsbuild test
```

## Dev environment

We are using Vagrant to manage dev environments.

The goal of the Vagrant setup is to provide, in a simple 'black box' that just works™, all the services that you need to run hecuba.

You shouldn't need to login to the vagrant box at all, everything is forwarded outside the box.

You will still run your hecuba instance local to your machine (not in the vagrant box).


+ Install VirtualBox v4.3.28 (correct as of 10th June 2015) from [here](https://www.virtualbox.org/wiki/Downloads) or preferably via your OS's package manager.
+ Install Vagrant v. 1.7.2 (correct as of 10th June 2015) from [here](http://www.vagrantup.com/) or preferably via your OS's package manager.
+ install the vbguest plugin so Virtual Box guest additions will updated
  for you ``vagrant plugin install vagrant-vbguest``
+ ``cd ${PROJECT_HOME}``
+ ``vagrant up`` (This will download stuff the first time and will be slow, after that it will be quicker)
+ You will now have all the services required running in a virtual machine with the ports forwarded for access from your local machine
+ You will need to install a cassandra _client_ only on your local machine. (You should not be running a cassandra locally as the ports will clash with those forwarded from the vagrant machine).

### First Time Test Data

+ Work out what the hostname you should use to connect to cassandra is. Look at the output of ``netstat -tln``, find the line that says something like ``aaa.bbb.ccc.ddd:9042``. (note this might be an ipv6 address depending on how your network is configured). This is the address to use in your hecuba config file and in the commands below.

On your machine (not the vagrant box):

+ create the test schema ``cqlsh aaa.bbb.ccc.ddd -f hecuba-schema.sql`` (this might show an error the first time you run it about the test namespace not existing, you can ignore that error)
+ Start cqlsh with the right hostname ``cqlsh aaa.bbb.ccc.ddd``
+ Create the Elastic Search schema by running ``scripts/build-es-schema.sh``
+ Create a file ``~/.hecuba.edn``` with the following contents:
```
{
 :cassandra-session {:keyspace :test}
 :hecuba-session {:keyspace :test}
 :search-session {:host :host "<the address you found above>" :name "hecuba"}
 :s3          {:access-key "<your personal AWS access key DO NOT SHARE KEYS!>"
               :secret-key "<your personal AWS secret key DO NOT SHARE KEYS!>"
               ;; you will need to create these buckets.
               :file-bucket "mc-<yourname>-hecuba-uploads"
               :status-bucket "mc-<yourname>-hecuba-status"
               :download-dir "/tmp"}
               
}
```

Back on your host machine do the following:

+ Start a repl in your favourite way.
+ Start the application with (go)
+ (require 'etl)
+ (kixi.hecuba.security/add-user! (:store kixipipe.application/system) "Mastodon" "<EMAIL>" "password" :kixi.hecuba.security/super-admin  #{} #{} )
{'fraction_non_alphanumeric': 0.10245614035087719, 'fraction_numerical': 0.005847953216374269, 'mean_word_length': 3.914367816091954, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '156889', 'n_tokens_mistral': 2729, 'n_tokens_neox': 2562, 'n_words': 1141}