<commit_before># DataCafe - Data Federation and Integration
A Dynamic Data Warehousing System

Apache Drill can be used to query Hadoop and HDFS in a very efficient way. The steps are as below. Refer to the linked pages for each step.

1. Configuring Mongo in EC2.

Connecting to the remote EC2/Mongo Instance
$ ssh -i "pradeeban.pem" ubuntu@ec2-54-166-90-207.compute-1.amazonaws.com

Copy files to the remote instance.

$ scp -i "pradeeban.pem" /home/pradeeban/gsoc2015/conf/clinical.csv ubuntu@ec2-54-157-62-174.compute-1.amazonaws.com:/home/ubuntu

$ scp -i "pradeeban.pem" /home/pradeeban/gsoc2015/conf/pathology.csv ubuntu@ec2-54-157-62-174.compute-1.amazonaws.com:/home/ubuntu


Importing the collections from the csv files

mongoimport --db pathology --collection pathologyData --type csv --headerline --file pathology.csv
mongoimport --db clinical --collection clinicalData --type csv --headerline --file clinical.csv


Executing mongo commands.

mongo

use pathology

db.pathologyData.find()


use clinical

db.clinicalData.find()


2. Configure Hadoop 

There may be some errors encountered when executing Hadoop with Java 8 with the default configurations. Further, formatting the file system will fix any error from Hadoop.

 $ $HADOOP_HOME/bin/hdfs namenode -format



3. Start Hadoop NameNode daemon and DataNode daemon -

$ $HADOOP_HOME/sbin/start-dfs.sh


Once you are done with the experiments, you may stop the daemons with

$ $HADOOP_HOME/sbin/stop-dfs.sh


4. Browse the web interface for the name node - http://localhost:50070/


5. Configure Drill

* Launch Drill in Embedded mode -

$ $DRILL_HOME/bin/drill-embedded 

6. Browse the web interface for Drill - http://localhost:8047/

7. Configure Hive

$ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

Configure Hive Metastore with MySQL.

8. Run Hive Metastore and Hive

$ $HIVE_HOME/bin/hive --service metastore &

Start HiveServer2

$ HIVE_HOME/bin/hiveserver2


9.  Configure and enable Storage Plugin for Hive in Drill

{
  "type": "hive",
  "enabled": true,
  "configProps": {
    "hive.metastore.uris": "thrift://localhost:9083",
    "hive.metastore.sasl.enabled": "false"
  }
}


10. Query Hive from Drill.

SELECT firstname,lastname FROM hive.`customers` 

 SELECT * FROM hive.`patients`
 SELECT * FROM hive.`slices`

 SELECT slices.sliceID, slices.slideBarCode, patients.patientID, patients.gender, patients.laterality
 FROM hive.`patients`, hive.`slices`
 WHERE patients.patientID = slices.patientID



11. Executing DataCafe
$ mvn clean install
$ java -classpath lib/datacafe-server-1.0-SNAPSHOT.jar:lib/*:conf/ edu.emory.bmi.datacafe.impl.main.Initiator


12. SCP to remote instance
$ ssh -i "pradeeban.pem" ubuntu@ec2-54-157-222-55.compute-1.amazonaws.com:/home/ubuntu/


13. Copy All to the EC2 Instance
$ scp -r -i "pradeeban.pem" . ubuntu@ec2-54-157-222-55.compute-1.amazonaws.com:/home/ubuntu/datacafe


14. Copy only the datacafe jar

$ scp -i "pradeeban.pem" lib/datacafe-server-1.0-SNAPSHOT.jar ubuntu@ec2-54-157-222-55.compute-1.amazonaws.com:/home/ubuntu/datacafe/lib
















API Guide
=========
API for Datacafe
Public facing interface of DataCafe.
CreateDatacafe/
> Create/Add data source<commit_msg>README improvements.<commit_after># DataCafe - Data Federation and Integration
A Dynamic Data Warehousing System

Apache Drill can be used to query Hadoop and HDFS in a very efficient way. The steps are as below. Refer to the linked pages for each step.

1. Configuring Mongo in EC2.

Connecting to the remote EC2/Mongo Instance
$ ssh -i "pradeeban.pem" ubuntu@ec2-54-166-90-207.compute-1.amazonaws.com

Copy files to the remote instance.

$ scp -i "pradeeban.pem" /home/pradeeban/gsoc2015/conf/clinical.csv ubuntu@ec2-54-157-62-174.compute-1.amazonaws.com:/home/ubuntu

$ scp -i "pradeeban.pem" /home/pradeeban/gsoc2015/conf/pathology.csv ubuntu@ec2-54-157-62-174.compute-1.amazonaws.com:/home/ubuntu


Importing the collections from the csv files

mongoimport --db pathology --collection pathologyData --type csv --headerline --file pathology.csv
mongoimport --db clinical --collection clinicalData --type csv --headerline --file clinical.csv


Executing mongo commands.

mongo

use pathology

db.pathologyData.find()


use clinical

db.clinicalData.find()


2. Configure Hadoop 

There may be some errors encountered when executing Hadoop with Java 8 with the default configurations. Further, formatting the file system will fix any error from Hadoop.

 $ $HADOOP_HOME/bin/hdfs namenode -format



3. Start Hadoop NameNode daemon and DataNode daemon -

$ $HADOOP_HOME/sbin/start-dfs.sh


Once you are done with the experiments, you may stop the daemons with

$ $HADOOP_HOME/sbin/stop-dfs.sh


4. Browse the web interface for the name node - http://localhost:50070/


5. Configure Drill

* Launch Drill in Embedded mode -

$ $DRILL_HOME/bin/drill-embedded 


* Launch Drill during EMR instance bootstrap action
s3://maprtech-emr/scripts/mapr_drill_bootstrap.sh


6. Browse the web interface for Drill - http://localhost:8047/

7. Configure Hive

$ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

Configure Hive Metastore with MySQL.

8. Run Hive Metastore and Hive

$ $HIVE_HOME/bin/hive --service metastore &

Start HiveServer2

$ HIVE_HOME/bin/hiveserver2


9.  Configure and enable Storage Plugin for Hive in Drill

{
  "type": "hive",
  "enabled": true,
  "configProps": {
    "hive.metastore.uris": "thrift://localhost:9083",
    "hive.metastore.sasl.enabled": "false"
  }
}


10. Query Hive from Drill.

SELECT firstname,lastname FROM hive.`customers` 

 SELECT * FROM hive.`patients`
 SELECT * FROM hive.`slices`

 SELECT slices.sliceID, slices.slideBarCode, patients.patientID, patients.gender, patients.laterality
 FROM hive.`patients`, hive.`slices`
 WHERE patients.patientID = slices.patientID



11. Executing DataCafe
$ mvn clean install
$ java -classpath lib/datacafe-server-1.0-SNAPSHOT.jar:lib/*:conf/ edu.emory.bmi.datacafe.impl.main.Initiator


12. SCP to remote instance
$ ssh -i "pradeeban.pem" ubuntu@ec2-54-157-222-55.compute-1.amazonaws.com:/home/ubuntu/


13. Copy All to the EC2 Instance
$ scp -r -i "pradeeban.pem" . ubuntu@ec2-54-157-222-55.compute-1.amazonaws.com:/home/ubuntu/datacafe


14. Copy only the datacafe jar

$ scp -i "pradeeban.pem" lib/datacafe-server-1.0-SNAPSHOT.jar ubuntu@ec2-54-157-222-55.compute-1.amazonaws.com:/home/ubuntu/datacafe/lib


Current AWS Deployment:
1 node for Mongo.
1 node for Java - datacafe.
1 node for Drill + EMR Hadoop/Hive master.
2 nodes for EMR Hadoop/Hive core. (Can be more than 2, with no code change. Currently 0 Task instances.
Task instances can be introduced without code changes as well).

Currently, 5 EC2 nodes in total.












API Guide
=========
API for Datacafe
Public facing interface of DataCafe.
CreateDatacafe/
> Create/Add data source
{'fraction_non_alphanumeric': 0.10567673756943503, 'fraction_numerical': 0.03427719821162444, 'mean_word_length': 3.6691967109424417, 'pattern_counts': {'":': 10, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 12, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2666300', 'n_tokens_mistral': 2905, 'n_tokens_neox': 2610, 'n_words': 738}