<commit_before># BigDawg Middleware

manydbbench
===========

If you hit a maven bug on [1.7, not closed
in maven repository (~/.m2/repository) edit
 org/glassfish/jersey/project/2.19/project-2.19.pom
 to have [1.7,) instead of [1.7,

Might need maven3 on ubuntu 14+. See http://sysads.co.uk/2014/05/install-apache-maven-3-2-1-ubuntu-14-04/

Requires
----------
Java 8

Mvn -- version 3

PostgreSQL >= 9.4

SciDB 14.12

Accumulo 1.7.1

Eclipse GUI guide
-------------------

Prepare:
-- Import with Eclipse, then right click the project 'istc.bigdawg [...]' -> property -> Maven -> Active Maven Profile -> put down the word 'dev';

Run: 
-- follow the Phase specific guides to set up the databases, make sure you could access each of which BigDAWG uses  
-- In Eclipse, right click Main.java under istc.bigdawg, and do "Run as" -> "Java Application" (or on OS X press "shift + fn + command + F11")



Command Line guide
-----------------

Prepare Eclipse (may need eclipse restart)
-> mvn -Declipse.workspace=[path to workspace] eclipse:add-maven-repo
-> mvn eclipse:eclipse


download dependencies
-> mvn install -P XXX  
  (note: XXX is the word 'dev', 'test', or 'prod') 

build jar
-> mvn package -P XXX

run test server
-- follow the Phase specific guides to set up the databases, make sure you could access each of which BigDAWG uses
-> mvn exec:java -P XXX

cmigrator
-----------------
Our main migration tool was built in C++. Please, refer to: src/main/cmigrator/README.md for more details.

# For demo of Phase 0.3
Migration from PostgreSQL to PostgreSQL is in binary format.

Migration between PostgreSQL and SciDB was implemented in CSV as well as in binary formats.

It occurs that the binary migration from PostgreSQL to SciDB should be faster in binary format. This is mainly because the binary loading to SciDB does not contain any intermediary transformation while the CSV loading to SciDB contains the costly transformation from CSV to an array format.

On the other hand, there is an opposite situation for data migration from SciDB to PostgreSQL. It means that we can extract data from an array that is multi-dimensional with its dimensions in CSV format whereas we cannot save the array in the binary format with its dimensions. This means that a full binary export from SciDB with the dimensions would require a re-dimension and we know that it is costly. The numbers will appear soon.

The next step is to build a distributed migrator so that we will be able to migrate data between nodes in a cluster. Currently, this is possible only for the migration between PostgreSQL nodes (thanks to the very convenient copy command (load,export) in PostgreSQL).

# For demo of Phase 0.2:

The main question is how we can handle many database instances. The idea is to store the meta-information (instance host, port, datbase name, etc.) in the catalog. The only connection information stored in the config file: bigdawgmiddle/src/main/resources/bigdawg-config.properties pertain to the PostgreSQL instance where the data for the catalog is stored. Additionally, the config file: bigdawgmiddle/src/main/resources/bigdawg-log4j.properties contains information about PostgreSQL instance where the logs are stored. 


### Prepare environment with 2 PostgreSQL instances:
- go to bigdawgmiddle/installation (there are the scripts that we will use)
- mkdir data
- cd data
- download mimic2.pgd to the data directory from [https://app.box.com/s/8by2c36m8bwxl9654bwf3mttdt25uu4k](Link URL)
- run script: **bash setup.sh** (this script should be executed only once)
- to stop the PostgreSQL instances run: bash stop_postgres.sh
- to start the PostgreSQL instances run: bash start_postgres.sh

### Additionally, you can change pom.xml and try migrating data from postgres1 to postgres2:
- go to bigdawgmiddle/
- in pom.xml uncomment: <bigDawg.main.class>istc.bigdawg.migration.FromPostgresToPostgres</bigDawg.main.class>
- in pom.xml comment: <bigDawg.main.class>istc.bigdawg.Main</bigDawg.main.class>
- run: mvn clean compile -P dev
- run: mvn exec:java

```
#!bash

<commit_msg>Fix mimic2.pgd link in README.<commit_after># BigDawg Middleware

manydbbench
===========

If you hit a maven bug on [1.7, not closed
in maven repository (~/.m2/repository) edit
 org/glassfish/jersey/project/2.19/project-2.19.pom
 to have [1.7,) instead of [1.7,

Might need maven3 on ubuntu 14+. See http://sysads.co.uk/2014/05/install-apache-maven-3-2-1-ubuntu-14-04/

Requires
----------
Java 8

Mvn -- version 3

PostgreSQL >= 9.4

SciDB 14.12

Accumulo 1.7.1

Eclipse GUI guide
-------------------

Prepare:
-- Import with Eclipse, then right click the project 'istc.bigdawg [...]' -> property -> Maven -> Active Maven Profile -> put down the word 'dev';

Run: 
-- follow the Phase specific guides to set up the databases, make sure you could access each of which BigDAWG uses  
-- In Eclipse, right click Main.java under istc.bigdawg, and do "Run as" -> "Java Application" (or on OS X press "shift + fn + command + F11")



Command Line guide
-----------------

Prepare Eclipse (may need eclipse restart)
-> mvn -Declipse.workspace=[path to workspace] eclipse:add-maven-repo
-> mvn eclipse:eclipse


download dependencies
-> mvn install -P XXX  
  (note: XXX is the word 'dev', 'test', or 'prod') 

build jar
-> mvn package -P XXX

run test server
-- follow the Phase specific guides to set up the databases, make sure you could access each of which BigDAWG uses
-> mvn exec:java -P XXX

cmigrator
-----------------
Our main migration tool was built in C++. Please, refer to: src/main/cmigrator/README.md for more details.

# For demo of Phase 0.3
Migration from PostgreSQL to PostgreSQL is in binary format.

Migration between PostgreSQL and SciDB was implemented in CSV as well as in binary formats.

It occurs that the binary migration from PostgreSQL to SciDB should be faster in binary format. This is mainly because the binary loading to SciDB does not contain any intermediary transformation while the CSV loading to SciDB contains the costly transformation from CSV to an array format.

On the other hand, there is an opposite situation for data migration from SciDB to PostgreSQL. It means that we can extract data from an array that is multi-dimensional with its dimensions in CSV format whereas we cannot save the array in the binary format with its dimensions. This means that a full binary export from SciDB with the dimensions would require a re-dimension and we know that it is costly. The numbers will appear soon.

The next step is to build a distributed migrator so that we will be able to migrate data between nodes in a cluster. Currently, this is possible only for the migration between PostgreSQL nodes (thanks to the very convenient copy command (load,export) in PostgreSQL).

# For demo of Phase 0.2:

The main question is how we can handle many database instances. The idea is to store the meta-information (instance host, port, datbase name, etc.) in the catalog. The only connection information stored in the config file: bigdawgmiddle/src/main/resources/bigdawg-config.properties pertain to the PostgreSQL instance where the data for the catalog is stored. Additionally, the config file: bigdawgmiddle/src/main/resources/bigdawg-log4j.properties contains information about PostgreSQL instance where the logs are stored. 


### Prepare environment with 2 PostgreSQL instances:
- go to bigdawgmiddle/installation (there are the scripts that we will use)
- mkdir data
- cd data
- download mimic2.pgd to the data directory from [BitBucket](https://bitbucket.org/adam-dziedzic/bigdawgdata/raw/6ade22253695bfeb33074e82929e83b52cb121f1/mimic2.pgd) or [box.com](https://app.box.com/s/8by2c36m8bwxl9654bwf3mttdt25uu4k).
- run script: **bash setup.sh** (this script should be executed only once)
- to stop the PostgreSQL instances run: bash stop_postgres.sh
- to start the PostgreSQL instances run: bash start_postgres.sh

### Additionally, you can change pom.xml and try migrating data from postgres1 to postgres2:
- go to bigdawgmiddle/
- in pom.xml uncomment: <bigDawg.main.class>istc.bigdawg.migration.FromPostgresToPostgres</bigDawg.main.class>
- in pom.xml comment: <bigDawg.main.class>istc.bigdawg.Main</bigDawg.main.class>
- run: mvn clean compile -P dev
- run: mvn exec:java

```
#!bash


{'fraction_non_alphanumeric': 0.08922335942203492, 'fraction_numerical': 0.018904274533413607, 'mean_word_length': 3.8459743290548425, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 33, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '432933', 'n_tokens_mistral': 2568, 'n_tokens_neox': 2395, 'n_words': 1105}