<commit_before>##Technologies
<strong>Database:</strong> MongoDB<br>
<strong>Framework:</strong> Rails 4<br>
<strong>Dependencies:</strong> [Rails-API gem](https://github.com/rails-api/rails-api)<br>
<strong>Testing:</strong> Rspec and Cucumber<br>
<strong>ORM:</strong> [MongoMapper](www.mongomapper.com "MongoMapper")<br>

###Getting Started
1. Fork the repo
2. Clone your fork of un_data_api to your local computer
3. bundle install
4. Get Mongo running
5. Open up another tab in your terminal and change into the un_data_api directory

```
$ git clone https://github.com/username/un_data_api.git
$ cd un_data_api
$ bundle install
$ mongod

# open up another terminal window
$ cd un_data_api
$ rake db:create
$ rake xml_parser:seed_db
```
###Setting up with 3scale
Currently the UN Data API has the plugin implemented into the code already.  It only takes a few steps and you'll be up and running with authentication! In order to use 3scale you need to signup and retrieve the keys you need to use your API.

1. Go to www.3scale.net
2. Click on the big orange button "SIGNUP NOW-FREE!"
3. Sign up!
4. Go to your admin portal and log in. https://your-api-name-admin.3scale.net .
5. Set up your environment variables for development.

#####Enviroment Variables
**Remember the enviroment variables and keys are secret.  Make sure that your .gitignore is configured properly so your keys do not end up on github.  Here is a good article on how to implement environment variables. http://railsapps.github.io/rails-environment-variables.html
im
You can look at the example local environment yaml file to see how to format your file. 
```yml
# config/examples/local_env.yml
PROVIDER_KEY: 'some_provider_key'
APP_ID: 'some_app_id'
APP_KEY: 'some_app_key'
```
#####Active Docs
***You will not be able to use this feature until you deploy into production.
Lucky for you, I have already written and saved the file that generates the active docs. Here are 3scale's docs for Active Docs https://support.3scale.net/howtos/api-configuration#ConfigureActiveDocs .

1. Go to your admin portal.
2. Click on "API"
3. Click on "Active Docs"
4. Click "Create a new spec"
5. Give the spec a name and system_name
6. Get the spec. The only thing you will need to change is the "basePath" to the base url of your project. Go to un_data_active_doc_spec.json 

```json
{
  "basePath": "http://your-api.org/",
  "apiVersion": "v1",
  "apis": [
    {
```
7. You will have to click "publish" if you want them to show up on your Developer Portal.

###Set up without 3scale
If you would like to remove authentication you just need to remove or comment out the following methods from the Application Controller
#####Remove :authenticate_app from the before_filter

```ruby
before_filter :authenticate_app, :default_format_json, :set_access_control_headers
```
#####Remove create_client

```ruby
def create_client
  @@threescale_client ||= ThreeScale::Client.new(:provider_key => ENV['PROVIDER_KEY'])    
end
```

#####Remove authenticate_app

```ruby
def authenticate_app
  response = create_client.authrep(:app_id => params["app_id"], 
                                   :app_key => params["app_key"],
                                   :usage => { params[:metric].to_sym => 1 })
  response_success(response)
end
```

#####Remove response_success(response)

```ruby
def response_success(response)
  if response.success?
    return true
  else
    error(response.error_code, 401, response.error_message)
  end
end
```

###Deploy on Heroku
set your heroku config variables
provider_key
app_id
app_key

  
##Backup your database on Heroku

If you deploy on heroku the project already has the [heroku-mongo-backup gem](http://rubydoc.info/gems/heroku-mongo-backup/0.4.32/frames) installed. 

```
heroku config:add S3_BACKUPS_BUCKET=_value_ S3_KEY_ID=_value_ S3_SECRET_KEY=_value_ MONGO_URL=_value_
```

Example command to back up the database...

```
heroku run rake mongo:backup
```

##XML Parser Documentation
###How to add new databases
####Example Database: World Development Indicators
Go to http://data.un.org/Explorer.aspx and pick a database that you would like to import. For the purposes of this example we are going to use the database World Development Indicators and go throught the steps of how I added this database. The first two items you will need to take note of are the **database name** and the **organization name**.

- **Organization Name:** "World Bank"
- **Organization Acronym** "WB"
- **Full Database Name:** "World Development Indicators"

SCREEN SHOT
![Image](data-un-name-screenshot.png?raw=true)
<img src="data-un-name-screenshot.png"></img>

Next you will have to download all of the datasets that are under the database. Your file structure should match how the files are listed on that data.un.org website. In “lib/un_data_xml_files/” you will create a directory that is called the acronym of the organzation that the database belongs to. There are multiple databases in an organization so you should check if the directory already exists for the organization. Within the organzation's directory you will create a new directory named the full name of the new database. You will then download all the datasets and save them under this directory.

1. Next to the dataset click "view data". When you click through each dataset you should take note if they have footnotes.- **SCREEN SHOT**
2. Click on download and then a box will pop up with the different options to download the dataset. **SCREEN SHOT**
3. Select the XML option **SCREEN SHOT**
4. When the file downloads it will be in a zip format and the file name is a random name assigned by the data.un.org site. **SCREEN SHOT**
5. You will have to unzip the file. **SCREEN SHOT**
6. Copy the dataset name from the data.un.org website **SCREEN SHOT**
7. rename the dataset file to the same as displayed on the website. We try to keep the names exactly the same but sometimes the names have characters that are not supported in filenames.  In that case you will have to replace the character with the appropriate word. **SCREEN SHOT**
**show example of a nested directory

####Step 1: Set Up Rake Tasks
To create the rake tasks that you will use to populate the database you will need a few more pieces of information. 

#####Database Acronym
- Find the name of the database you will be importing. 
- Make sure it is the name of the database and not the name of the organization.
- Get the database acronym
- **Database Acronym:** "wdi"
```
*database_acronym = a lowercase acronym of the database name
```
- Checkout a new branch with parse_ + database_acronym<commit_msg>Test adding images<commit_after>##Technologies
<strong>Database:</strong> MongoDB<br>
<strong>Framework:</strong> Rails 4<br>
<strong>Dependencies:</strong> [Rails-API gem](https://github.com/rails-api/rails-api)<br>
<strong>Testing:</strong> Rspec and Cucumber<br>
<strong>ORM:</strong> [MongoMapper](www.mongomapper.com "MongoMapper")<br>

###Getting Started
1. Fork the repo
2. Clone your fork of un_data_api to your local computer
3. bundle install
4. Get Mongo running
5. Open up another tab in your terminal and change into the un_data_api directory

```
$ git clone https://github.com/username/un_data_api.git
$ cd un_data_api
$ bundle install
$ mongod

# open up another terminal window
$ cd un_data_api
$ rake db:create
$ rake xml_parser:seed_db
```
###Setting up with 3scale
Currently the UN Data API has the plugin implemented into the code already.  It only takes a few steps and you'll be up and running with authentication! In order to use 3scale you need to signup and retrieve the keys you need to use your API.

1. Go to www.3scale.net
2. Click on the big orange button "SIGNUP NOW-FREE!"
3. Sign up!
4. Go to your admin portal and log in. https://your-api-name-admin.3scale.net .
5. Set up your environment variables for development.

#####Enviroment Variables
**Remember the enviroment variables and keys are secret.  Make sure that your .gitignore is configured properly so your keys do not end up on github.  Here is a good article on how to implement environment variables. http://railsapps.github.io/rails-environment-variables.html
im
You can look at the example local environment yaml file to see how to format your file. 
```yml
# config/examples/local_env.yml
PROVIDER_KEY: 'some_provider_key'
APP_ID: 'some_app_id'
APP_KEY: 'some_app_key'
```
#####Active Docs
***You will not be able to use this feature until you deploy into production.
Lucky for you, I have already written and saved the file that generates the active docs. Here are 3scale's docs for Active Docs https://support.3scale.net/howtos/api-configuration#ConfigureActiveDocs .

1. Go to your admin portal.
2. Click on "API"
3. Click on "Active Docs"
4. Click "Create a new spec"
5. Give the spec a name and system_name
6. Get the spec. The only thing you will need to change is the "basePath" to the base url of your project. Go to un_data_active_doc_spec.json 

```json
{
  "basePath": "http://your-api.org/",
  "apiVersion": "v1",
  "apis": [
    {
```
7. You will have to click "publish" if you want them to show up on your Developer Portal.

###Set up without 3scale
If you would like to remove authentication you just need to remove or comment out the following methods from the Application Controller
#####Remove :authenticate_app from the before_filter

```ruby
before_filter :authenticate_app, :default_format_json, :set_access_control_headers
```
#####Remove create_client

```ruby
def create_client
  @@threescale_client ||= ThreeScale::Client.new(:provider_key => ENV['PROVIDER_KEY'])    
end
```

#####Remove authenticate_app

```ruby
def authenticate_app
  response = create_client.authrep(:app_id => params["app_id"], 
                                   :app_key => params["app_key"],
                                   :usage => { params[:metric].to_sym => 1 })
  response_success(response)
end
```

#####Remove response_success(response)

```ruby
def response_success(response)
  if response.success?
    return true
  else
    error(response.error_code, 401, response.error_message)
  end
end
```

###Deploy on Heroku
set your heroku config variables
provider_key
app_id
app_key

  
##Backup your database on Heroku

If you deploy on heroku the project already has the [heroku-mongo-backup gem](http://rubydoc.info/gems/heroku-mongo-backup/0.4.32/frames) installed. 

```
heroku config:add S3_BACKUPS_BUCKET=_value_ S3_KEY_ID=_value_ S3_SECRET_KEY=_value_ MONGO_URL=_value_
```

Example command to back up the database...

```
heroku run rake mongo:backup
```

##XML Parser Documentation
###How to add new databases
####Example Database: World Development Indicators
Go to http://data.un.org/Explorer.aspx and pick a database that you would like to import. For the purposes of this example we are going to use the database World Development Indicators and go throught the steps of how I added this database. The first two items you will need to take note of are the **database name** and the **organization name**.

- **Organization Name:** "World Bank"
- **Organization Acronym** "WB"
- **Full Database Name:** "World Development Indicators"

SCREEN SHOT
![Image](https://github.com/3scale/un_data_api/raw/master/public/images/data-un-name-screenshot.png)

Next you will have to download all of the datasets that are under the database. Your file structure should match how the files are listed on that data.un.org website. In “lib/un_data_xml_files/” you will create a directory that is called the acronym of the organzation that the database belongs to. There are multiple databases in an organization so you should check if the directory already exists for the organization. Within the organzation's directory you will create a new directory named the full name of the new database. You will then download all the datasets and save them under this directory.

1. Next to the dataset click "view data". When you click through each dataset you should take note if they have footnotes.- **SCREEN SHOT**
2. Click on download and then a box will pop up with the different options to download the dataset. **SCREEN SHOT**
3. Select the XML option **SCREEN SHOT**
4. When the file downloads it will be in a zip format and the file name is a random name assigned by the data.un.org site. **SCREEN SHOT**
5. You will have to unzip the file. **SCREEN SHOT**
6. Copy the dataset name from the data.un.org website **SCREEN SHOT**
7. rename the dataset file to the same as displayed on the website. We try to keep the names exactly the same but sometimes the names have characters that are not supported in filenames.  In that case you will have to replace the character with the appropriate word. **SCREEN SHOT**
**show example of a nested directory

####Step 1: Set Up Rake Tasks
To create the rake tasks that you will use to populate the database you will need a few more pieces of information. 

#####Database Acronym
- Find the name of the database you will be importing. 
- Make sure it is the name of the database and not the name of the organization.
- Get the database acronym
- **Database Acronym:** "wdi"
```
*database_acronym = a lowercase acronym of the database name
```
- Checkout a new branch with parse_ + database_acronym
{'fraction_non_alphanumeric': 0.08633039282213677, 'fraction_numerical': 0.0068611927919776826, 'mean_word_length': 4.318364073777065, 'pattern_counts': {'":': 6, '<': 35, '<?xml version=': 0, '>': 45, 'https://': 9, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '152609', 'n_tokens_mistral': 3921, 'n_tokens_neox': 3766, 'n_words': 1765}