<commit_before>This is the Design and Implementation document detailing how BukGet is constructed and implemented on the current servers.  This document will be updated as the current implementation changes and is designed to be a guide for individuals looking to assist in maintaining the BukGet API.

## Arch Overview

BukGet utilizes MongoDB as the back-end database and is coded using the bottlepy web framework.  For scalability, we use Paste as the application server and front-end that with Nginx for URL routing (in-case we need to stand up a dev instance) as well a logging all of the requests.  This implementation framework has been able to handle 20M requests a month without showing any significant impact on the hardware (~10% CPU Utilization and ~1-2% I/OWait as of the writing of this document).

__Application List__

* MongoDB
* Python 2.6/7
* Bottle
* Paste
* Nginx

__Script List__

* bukgen_bukkit
* bukgen_manual
* bukget
* force_update.py
* logreader.py

__ Services List__

* nginx - sysv
* mongod - sysv
* bukget - upstart


### MongoDB

The Mongo Database has not had any customization that needs to be documented at this time.  We will eventually be looking at going multi-server, and will be using MongoDB's replication capabilities for distributing the database across all of the servers.


### Python 2.6/7

The Python Installation on the server(s) is the default installation that comes with CentOS 6.3 x64.  Pip has also been installed on these hosts manually and is being used to help manage the installation of any needed dependencies as well as the installation of the bukget and bukgen packages themselves.  These packages are being pulled directly from the git repository and then being updated using the following command in the appropriate directory:

`pip install --upgrade ./`

This will install/upgrade any dependencies for the package as well as install the latest version of the package itself.


### Nginx

Below is the configuration for the VHost that we use for BukGet.  Keep in mind that minor modifications may exist to also listen for the hostname as well as api.bukget.org

	server {
	        listen          80;
	        server_name     api.bukget.org XXX.api.bukget.org;

	        access_log      /var/log/bukget/api-access.log;
	        error_log       /var/log/bukget/api-error.log;

	        location / {
	                proxy_pass              http://127.0.0.1:9132/;
	                proxy_redirect          off;

	                proxy_set_header                Host            $host;
	                proxy_set_header                X-Real-IP       $remote_addr;
	                proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
	                proxy_max_temp_file_size        0;
<commit_msg>Updated the Design doc to include server breakdown.<commit_after>This is the Design and Implementation document detailing how BukGet is constructed and implemented on the current servers.  This document will be updated as the current implementation changes and is designed to be a guide for individuals looking to assist in maintaining the BukGet API.

## Arch Overview

BukGet utilizes MongoDB as the back-end database and is coded using the bottlepy web framework.  For scalability, we use Paste as the application server and front-end that with Nginx for URL routing (in-case we need to stand up a dev instance) as well a logging all of the requests.  This implementation framework has been able to handle 20M requests a month without showing any significant impact on the hardware (~10% CPU Utilization and ~1-2% I/OWait as of the writing of this document).

__Application List__

* MongoDB
* Python 2.6/7
* Bottle
* Paste
* Nginx

__Script List__

* bukgen_bukkit
* bukgen_manual
* bukget
* force_update.py
* logreader.py

__Services List__

* nginx - sysv
* mongod - sysv
* bukget - upstart


__Services Layout (By Server)__

* dallas.api.bukget.org
    * BukGet API (Nginx, Mongo, BukGet)
    * logreader script (cronjob)
    * BukGen Generator (bukgen_* applications)
    * Maintinence Functions (force_update.py, bukgen_manual)
* paris.api.bukget.org
    * BukGet API (Nginx, Mongo, BukGet)
* dev.vpn.bukget.org
    * Development Services (Independent of Prod)
        * BukGet API (Nginx, Mongo, BukGet)
    * Production Database Nightly Backup


### MongoDB

The Mongo Database has not had any customization that needs to be documented at this time.  The Dallas server is the primary database engine, with the Paris server acting as the secondary.  This means that all Database writes has to go through the Dallas Mongo instance.


### Python 2.6/7

The Python Installation on the server(s) is the default installation that comes with CentOS 6.3 x64.  Pip has also been installed on these hosts manually and is being used to help manage the installation of any needed dependencies as well as the installation of the bukget and bukgen packages themselves.  These packages are being pulled directly from the git repository and then being updated using the following command in the appropriate directory:

`pip install --upgrade ./`

This will install/upgrade any dependencies for the package as well as install the latest version of the package itself.


### Nginx

Below is the configuration for the VHost that we use for BukGet.  Keep in mind that minor modifications may exist to also listen for the hostname as well as api.bukget.org

	server {
	        listen          80;
	        server_name     api.bukget.org XXX.api.bukget.org;

	        access_log      /var/log/bukget/api-access.log;
	        error_log       /var/log/bukget/api-error.log;

	        location / {
	                proxy_pass              http://127.0.0.1:9132/;
	                proxy_redirect          off;

	                proxy_set_header                Host            $host;
	                proxy_set_header                X-Real-IP       $remote_addr;
	                proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
	                proxy_max_temp_file_size        0;

{'fraction_non_alphanumeric': 0.047587683187880786, 'fraction_numerical': 0.009550469290301334, 'mean_word_length': 2.8712555768005097, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7007068', 'n_tokens_mistral': 1717, 'n_tokens_neox': 1566, 'n_words': 747}