<commit_before>Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently wrap the API operations in a manner that makes working with EMR job flows from Ruby more enjoyable.  At the very least, using Elasticity allows you to easily experiment with the EMR API :)

'''BACKLOG''': Have a look at the [https://www.pivotaltracker.com/projects/272429 backlog] to see where this is headed.

'''CREDITS''': AWS signing was used from [http://www.rightscale.com/ RightScale's] amazing [https://github.com/rightscale/right_aws right_aws gem] which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.  Used camelize from ActiveSupport as well, thank you \Rails :)

= Installation and Usage =

<pre>
  gem install elasticity
</pre>

All you have to do is <code>require 'elasticity'</code> and you're all set!

= API Reference =

Elasticity wraps all of the EMR API calls.  Please see the Amazon guide for details on these operations because the default values aren't obvious (e.g. the meaning of <code>DescribeJobFlows</code> without parameters).

You may opt for "direct" access to the API where you specify the params and Elasticity takes care of the signing for you, responding with the XML from Amazon.  Direct access is described below the API catalog.

In addition to the [http://aws.amazon.com/elasticmapreduce/ AWS EMR subsite], there are three primary resources of reference information for EMR:

* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/ Amazon EMR Getting Started Guide]
* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/ Amazon EMR Developer Guide]
* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/ Amazon EMR API Reference]

Unfortunately, the documentation is sometimes incorrect and sometimes missing.  E.g. the allowable values for AddInstanceGroups are present in the [http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf PDF] version of the API reference but not in the [http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/ HTML] version.  Elasticity implements the API as specified in the PDF reference as that is the most complete description I could find.

== AddInstanceGroups ==

AddInstanceGroups adds a group of instances to an existing job flow.  The available instance configuration options are listed in the EMR API reference.  They've been converted to be more Ruby-like in the wrappers, as shown in the example below.

<pre>
  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
  instance_group_config = {
    :instance_count => 1,
    :instance_role => "TASK",
    :instance_type => "m1.small",
    :market => "ON_DEMAND",
    :name => "Go Canucks Go!"
  }
  emr.add_instance_groups("j-26LIXPUNSC0M3", [instance_group_config])
  <commit_msg>Added docs for HiveJob.<commit_after>Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently wrap the API operations in a manner that makes working with EMR job flows from Ruby more productive and more enjoyable, without having to understand the nuts and bolts of the EMR REST API.  At the very least, using Elasticity allows you to easily experiment with the EMR API :)

'''BACKLOG''': Have a look at the [https://www.pivotaltracker.com/projects/272429 backlog] to see where this is headed.

'''CREDITS''': AWS signing was used from [http://www.rightscale.com/ RightScale's] amazing [https://github.com/rightscale/right_aws right_aws gem] which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.  Used camelize from ActiveSupport as well, thank you \Rails :)

= Installation and Usage =

<pre>
  gem install elasticity
</pre>

All you have to do is <code>require 'elasticity'</code> and you're all set!

= Simplified API Reference =

Elasticity currently provides simplified access to launching Hive job flows.  Eventually I'd like to add several simplified wrappers, but I only need Hive at the moment :)

== Hive ==

HiveJob allows you to quickly launch Hive jobs without having to understand the ins and outs of the EMR API.  Specify only the Hive script location and (optionally) variables to make available to the Hive script.

<pre>
  hive = Elasticity::HiveJob.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
  hive.run("s3n://slif-hive/test.q", {
    "LIB"    => "s3n://slif-test/lib",
    "OUTPUT" => "s3n://slif-test/output"
  })
  
  > "j-129V5AQFMKO1C"
</pre>

= Amazon API Reference =

Elasticity wraps all of the EMR API calls.  Please see the Amazon guide for details on these operations because the default values aren't obvious (e.g. the meaning of <code>DescribeJobFlows</code> without parameters).

You may opt for "direct" access to the API where you specify the params and Elasticity takes care of the signing for you, responding with the XML from Amazon.  Direct access is described below the API catalog.

In addition to the [http://aws.amazon.com/elasticmapreduce/ AWS EMR subsite], there are three primary resources of reference information for EMR:

* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/ Amazon EMR Getting Started Guide]
* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/ Amazon EMR Developer Guide]
* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/ Amazon EMR API Reference]

Unfortunately, the documentation is sometimes incorrect and sometimes missing.  E.g. the allowable values for AddInstanceGroups are present in the [http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf PDF] version of the API reference but not in the [http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/ HTML] version.  Elasticity implements the API as specified in the PDF reference as that is the most complete description I could find.

== AddInstanceGroups ==

AddInstanceGroups adds a group of instances to an existing job flow.  The available instance configuration options are listed in the EMR API reference.  They've been converted to be more Ruby-like in the wrappers, as shown in the example below.

<pre>
  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
  instance_group_config = {
    :instance_count => 1,
    :instance_role => "TASK",
    :instance_type => "m1.small",
    :market => "ON_DEMAND",
    :name => "Go Canucks Go!"
  }
  emr.add_instance_groups("j-26LIXPUNSC0M3", [instance_group_config])
  
{'fraction_non_alphanumeric': 0.08581862446743761, 'fraction_numerical': 0.010651247717589775, 'mean_word_length': 4.6131511528608025, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 32, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6225503', 'n_tokens_mistral': 1962, 'n_tokens_neox': 1880, 'n_words': 771}