<commit_before>BigData_for_Energy_Efficiency
=============================
### !!!Under Construction!!!####

This project is an endeavour to provide a proof of concept for using Big Data along with advanced analytics techniques to solve issues for improving energy efficiency. In this project we used some sample data sets provided by "Technical Research Centre of Finland - VTT". These data sets were collected using specialized metering devices installed by VTT in a group of buildings. For sake of of anonymity the names and locations of the buildings are replaced by hypothetical labels. Following is a list of energy effeciency use cases that were considered as part of this proof of concept.

1. Understanding the seasonal energy usage patterns and their sensitivity with outside temperature.
2. Understanding characteristics of the buildings using daily energy consumption pattern.
3. Calculating the base load of the building to identify non user consumption of buildings.
4. Classifying the buildings on the basis of the energy efficiency and analyse the seasonal shifts in this classification.
5. Predicting daily energy consumption of various household devices on the basis of previous consumption pattern.
6. Collecting Twitter data for energy related topics using keyword filtering. 



Following is the list of tools used for Big Data, advanced analytics and results visualization.

* Apache Hadoop
* Apache Pig 
* Apache Flume
* Apache Hive
* Apache oozie
* Cloudera Impala
* R Programming for Statistical Computing
* Weka 
* Tableau Public 

We used Cloudera CDH4 for Apache Hadoop, Pig, Hive, Flume, Ozzie and Cloudera Imapala.

## Software Architecture

![alt tag](https://github.com/hazznain/BigData_for_Energy_Efficiency/blob/master/images/iplatform.png)

##Implementation
###Cloudera CDH4
For this proof of concept or any demo use Cloudera CDH4 quick virtual machine can be used that is available via [Cloudera official website for Quick VM ](http://www.cloudera.com/content/support/en/downloads/quickstart_vms/cdh-4-7-x.html).

For using the concept presented in this proof concept in a production environment, We recommend that you install Cloudera CDH 4.0 in pseudo distributed single node or multi node modes. We found this very helpful step by step instruction of installing CDH4 published by <NAME> [Step by step guide for installing and configuring CDH4](https://docs.google.com/file/d/0Bx6N95pJhrROblJiaEJ0dHpwVmc/edit). Detailed cloudera documentation are official via their official documentation pages,[Ways to install CDH4](http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_4_2.html).


<commit_msg>changes<commit_after>BigData_for_Energy_Efficiency
=============================
### !!!Under Construction!!!####

This project is an endeavour to provide a proof of concept for using Big Data along with advanced analytics techniques to solve issues for improving energy efficiency. In this project we used some sample data sets provided by "Technical Research Centre of Finland - VTT". These data sets were collected using specialized metering devices installed by VTT in a group of buildings. For sake of of anonymity the names and locations of the buildings are replaced by hypothetical labels. Following is a list of energy effeciency use cases that were considered as part of this proof of concept.

1. Understanding the seasonal energy usage patterns and their sensitivity with outside temperature.
2. Understanding characteristics of the buildings using daily energy consumption pattern.
3. Calculating the base load of the building to identify non user consumption of buildings.
4. Classifying the buildings on the basis of the energy efficiency and analyse the seasonal shifts in this classification.
5. Predicting daily energy consumption of various household devices on the basis of previous consumption pattern.
6. Collecting Twitter data for energy related topics using keyword filtering. 

Following is the list of tools used for Big Data, advanced analytics and results visualization.

* Apache Hadoop
* Apache Pig 
* Apache Flume
* Apache Hive
* Apache oozie
* Cloudera Impala
* R Programming for Statistical Computing
* Weka 
* Tableau Public 

We used Cloudera CDH4 for Apache Hadoop, Pig, Hive, Flume, Ozzie and Cloudera Imapala.

## Software Architecture

![alt tag](https://github.com/hazznain/BigData_for_Energy_Efficiency/blob/master/images/iplatform.png)

##Implementation
###Cloudera CDH4
For this proof of concept or any demo use Cloudera CDH4 quick virtual machine can be used that is available via [Cloudera official website for Quick VM ](http://www.cloudera.com/content/support/en/downloads/quickstart_vms/cdh-4-7-x.html).

For using the concept presented in this proof concept in a production environment, We recommend that you install Cloudera CDH 4.0 in pseudo distributed single node or multi node modes. We found this very helpful step by step instruction of installing CDH4 published by <NAME> [Step by step guide for installing and configuring CDH4](https://docs.google.com/file/d/0Bx6N95pJhrROblJiaEJ0dHpwVmc/edit). Detailed cloudera documentation are official via their official documentation pages,[Ways to install CDH4](http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_4_2.html).



{'fraction_non_alphanumeric': 0.06247675715879509, 'fraction_numerical': 0.010784678319077723, 'mean_word_length': 5.105561861520999, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6451470', 'n_tokens_mistral': 1449, 'n_tokens_neox': 1362, 'n_words': 672}