<commit_before># SnowPlow::Etl

## Introduction

SnowPlow::Etl is a Ruby gem (built with [Bundler] [bundler]) to run the daily
ETL (extract, transform, load) job which processes the CloudFront access logs
generated by `snowplow.js` and turns them into SnowPlow-format, 
day-partitioned event files, all stored in Amazon S3.

Currently the SnowPlow::Etl gem performs the ETL process on Amazon Elastic 
MapReduce using Hive. A future version of this gem will offer a second ETL 
option using a "vanilla Hadoop" process on EMR. If you want this alternative 
ETL option, please vote for [this issue] [hadoopetl].

## Installation

### Dependencies

To make use of SnowPlow::Etl you will need the following:

1. An [Amazon Web Services] [aws] account
2. SnowPlow tracking successfully implemented on your website. Please see the
   [Tracker Setup Guide] [trackerguide] for help implementing this
3. Git installed. Please see the [Git Installation Guide] [gitguide] as needed  
4. RubyGems installed. Please see the [RubyGems Installation Instructions] [gemsguide]
   as needed
5. Bundler (a Ruby gem) installed (`gem install bundler`)

### Installation

First checkout the repository (TODO: remove the third line when we merge this 
branch to master):

    $ git clone git://github.com/snowplow/snowplow.git
    $ cd snowplow
    $ git checkout etl-scripts
    
Now install the SnowPlow::ETL gem on your system:

    $ cd hive/snowplow-etl
    $ gem build snowplow-etl.gemspec 
    $ sudo gem install snowplow-etl-0.0.1.gem

Now test that the gem was installed successfully:

    $ bundle exec snowplow-etl --version
    snowplow-etl 0.0.1

Note that the `bundle exec` command will only work when you are inside the 
`snowplow-etl` folder.

### Configuration

SnowPlow::Etl requires a YAML format configuration file to run. There
is a configuration file template available in the repository at 
`hive/snowplow-etl/config/config.yml`. The contents should be as
follows:

    :aws:
      :access_key_id: ADD HERE
      :secret_access_key: ADD HERE
    :buckets:
      :query: ADD HERE
      :serde: ADD HERE
      :in: ADD HERE
      :out: ADD HERE
      :archive: ADD HERE
<commit_msg>Added bundle install instruction<commit_after># SnowPlow::Etl

## Introduction

SnowPlow::Etl is a Ruby gem (built with [Bundler] [bundler]) to run the daily
ETL (extract, transform, load) job which processes the CloudFront access logs
generated by `snowplow.js` and turns them into SnowPlow-format, 
day-partitioned event files, all stored in Amazon S3.

Currently the SnowPlow::Etl gem performs the ETL process on Amazon Elastic 
MapReduce using Hive. A future version of this gem will offer a second ETL 
option using a "vanilla Hadoop" process on EMR. If you want this alternative 
ETL option, please vote for [this issue] [hadoopetl].

## Installation

### Dependencies

To make use of SnowPlow::Etl you will need the following:

1. An [Amazon Web Services] [aws] account
2. SnowPlow tracking successfully implemented on your website. Please see the
   [Tracker Setup Guide] [trackerguide] for help implementing this
3. Git installed. Please see the [Git Installation Guide] [gitguide] as needed  
4. RubyGems installed. Please see the [RubyGems Installation Instructions] [gemsguide]
   as needed
5. Bundler (a Ruby gem) installed (`gem install bundler`)

### Installation

First checkout the repository (TODO: remove the third line when we merge this 
branch to master):

    $ git clone git://github.com/snowplow/snowplow.git
    $ cd snowplow
    $ git checkout etl-scripts
    
Now install the SnowPlow::ETL gem on your system:

    $ cd hive/snowplow-etl
    $ gem build snowplow-etl.gemspec 
    $ sudo gem install snowplow-etl-0.0.1.gem
    $ bundle install

Now test that the gem was installed successfully:

    $ bundle exec snowplow-etl --version
    snowplow-etl 0.0.1

Note that the `bundle exec` command will only work when you are inside the 
`snowplow-etl` folder.

### Configuration

SnowPlow::Etl requires a YAML format configuration file to run. There
is a configuration file template available in the repository at 
`hive/snowplow-etl/config/config.yml`. The contents should be as
follows:

    :aws:
      :access_key_id: ADD HERE
      :secret_access_key: ADD HERE
    :buckets:
      :query: ADD HERE
      :serde: ADD HERE
      :in: ADD HERE
      :out: ADD HERE
      :archive: ADD HERE

{'fraction_non_alphanumeric': 0.07207821737153251, 'fraction_numerical': 0.005457025920873124, 'mean_word_length': 3.714898177920686, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1178393', 'n_tokens_mistral': 1385, 'n_tokens_neox': 1324, 'n_words': 571}