<commit_before>



# gptools-for-aws



GP stands for [Geoprocessing](http://resources.arcgis.com/en/help/main/10.1/index.html#//002s00000001000000), and Geoprocessing is for everyone that uses [Esri's ArcGIS](http://www.esri.com/software/arcgis). The fundamental purpose of geoprocessing is to provide tools and a framework for performing analysis and managing your geographic data. ArcGIS provides a large suite of tools for performing GIS tasks that range from simple buffers and polygon overlays to complex regression analysis and image classification. Tools can be chained together feeding the output of one tool into another. You can also create custom tools and leverage them within this framework, such as the GP tools for AWS provided here in this project. 



The GP Tools for AWS are built to enable ArcGIS users to leverage Amazon Web Services through GP tools. This project leverages Elastic Map Reduce (EMR) the hosted hadoop framework, as well as, Simple Storage Service (S3) in AWS. In addition, the project leverages [GIS Tools for Hadoop](http://esri.github.io/gis-tools-for-hadoop/) to geo-enable hadoop.



## Features

* Start an EMR Cluster in AWS

* Execute a hive query language script on the EMR Cluster in AWS

* Get status on an EMR Cluster in AWS

* Terminate an EMR Cluster in AWS

* Upload a file to S3

* Download a file from S3





## Instructions



1. Fork and then clone the repo. 

2. Follow setup instructions

3. Run and try the samples.



Setup Instructions



The following steps describe the setup and configuration needed. It only needs to be done once, and then you re ready to use the GIS Hadoop on EMR GP Tools anytime you want.



<h3>

<a name="installboto" class="anchor" href="#installboto"/>

1. Installing boto</h3>



boto is the python package that is used to communicate with Amazon Web Services. It should be added first before the GP tools are used. This can easily be done using:

<br>

> pip install boto

<br>

For windows users to easily leverage pip for adding a python package, win-pip can be useful. You can download it from this link:

https://sites.google.com/site/pydatalog/python/pip-for-windows



Once win-pip is downloaded, it will need to be pointed to the Python interpreter used by ArcGIS Desktop. The default install location added by the ArcGIS install is C:\Python27\<ArcGIS version>\python.exe

If you have several directories under Python27, make sure to use the 32 bit directory, not the 64 bit. Also, make sure to use the directory that has the right version of Desktop that you re using.



Win-pip will add the pip module if it s not already there.



To add boto, type: pip install boto, then click Run.

<p>



<img src="/images/win-pip.png" alt="win-pip"  width="320"> 





<h3>

<a name="gptoolssetup" class="anchor" href="#gptoolssetup"/>

2. Adding the GP Tools For AWS</h3>



Download the zip file, and unzip it under a location on your disk. Add this folder as a new folder connection in your Catalog from ArcGIS Desktop. 



That s it!



You re now ready to use the GIS Hadoop on EMR GP Tools.





<h3>

<a name="gptoolsconfig" class="anchor" href="#gptoolsconfig"/>

3. Getting your Credentials ready to access your AWS account

</h3>



You will need to get the following parameters from your AWS account ready before using the tools.



A. To be able to execute the tools you will need an AWS account. Make sure you have your account credentials ready to add into the tools.



You can get your credentials by going to:



http://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.html

you will find it under Access keys (access key ID and secret access key)





B. Also, you will need a key-pair. This is a certificate to encrypt your account information when you try to access your instances over the Internet. 



You can get a key-pair created and downloaded through the following steps:



http://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.html

you will find it under Key Pairs
<commit_msg>update readme instructions<commit_after>



# gptools-for-aws



GP stands for [Geoprocessing](http://resources.arcgis.com/en/help/main/10.1/index.html#//002s00000001000000), and Geoprocessing is for everyone that uses [Esri's ArcGIS](http://www.esri.com/software/arcgis). The fundamental purpose of geoprocessing is to provide tools and a framework for performing analysis and managing your geographic data. ArcGIS provides a large suite of tools for performing GIS tasks that range from simple buffers and polygon overlays to complex regression analysis and image classification. Tools can be chained together feeding the output of one tool into another. You can also create custom tools and leverage them within this framework, such as the GP tools for AWS provided here in this project. 



The GP Tools for AWS are built to enable ArcGIS users to leverage Amazon Web Services through GP tools. This project leverages Elastic Map Reduce (EMR) the hosted hadoop framework, as well as, Simple Storage Service (S3) in AWS. In addition, the project leverages [GIS Tools for Hadoop](http://esri.github.io/gis-tools-for-hadoop/) to geo-enable hadoop.



## Features

* Start an EMR Cluster in AWS

* Execute a hive query language script on the EMR Cluster in AWS

* Get status on an EMR Cluster in AWS

* Terminate an EMR Cluster in AWS

* Upload a file to S3

* Download a file from S3





## Instructions



1. Fork and then clone the repo. 

2. Follow setup instructions

3. Run and try the samples.



Setup Instructions



The following steps describe the setup and configuration needed. It only needs to be done once, and then you re ready to use the GIS Hadoop on EMR GP Tools anytime you want.



<h3>

<a name="installboto" class="anchor" href="#installboto"/>

1. Installing boto</h3>



boto is the python package that is used to communicate with Amazon Web Services. It should be added first before the GP tools are used. This can easily be done using:

<br>

> pip install boto

<br>

For windows users to easily leverage pip for adding a python package, win-pip can be useful. You can download it from this link:

https://sites.google.com/site/pydatalog/python/pip-for-windows



Once win-pip is downloaded, <i>run it as an administrator</i>. Make sure to pointto the Python interpreter used by ArcGIS Desktop. The default install location added by the ArcGIS install is C:\Python27\<ArcGIS version>\python.exe

If you have several directories under Python27, make sure to use the 32 bit directory, not the 64 bit. Also, make sure to use the directory that has the right version of Desktop that you re using.



Win-pip will add the pip module if it s not already there.



To add boto, type: pip install boto, then click Run.

<p>



<img src="/images/win-pip.png" alt="win-pip"  width="320"> 





<h3>

<a name="gptoolssetup" class="anchor" href="#gptoolssetup"/>

2. Adding the GP Tools For AWS</h3>



Download the zip file, and unzip it under a location on your disk. Add this folder as a new folder connection in your Catalog from ArcGIS Desktop. 



That s it!



You re now ready to use the GIS Hadoop on EMR GP Tools.





<h3>

<a name="gptoolsconfig" class="anchor" href="#gptoolsconfig"/>

3. Getting your Credentials ready to access your AWS account

</h3>



You will need to get the following parameters from your AWS account ready before using the tools.



A. To be able to execute the tools you will need an AWS account. Make sure you have your account credentials ready to add into the tools.



You can get your credentials by going to:



http://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.html

you will find it under Access keys (access key ID and secret access key)





B. Also, you will need a key-pair. This is a certificate to encrypt your account information when you try to access your instances over the Internet. 



You can get a key-pair created and downloaded through the following steps:



http://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.html

you will find it under Key Pairs



<br>



C. Make sure your AWS account is enabled for the Elastic Map Reduce, EC2, and S3 Services. You can check that by seeing the list of services enabled for your account when you login. 

{'fraction_non_alphanumeric': 0.5294331615641104, 'fraction_numerical': 0.005698696574719612, 'mean_word_length': 8.524249422632794, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4454008', 'n_tokens_mistral': 16438, 'n_tokens_neox': 16437, 'n_words': 1221}