<commit_before># Docker, CoreOS, Weave, Kubernetes Multi-Host/Cloud Cluster

Setting up and, running a Docker cluster was a completely new and unknown area
of expertise for us. Our goal was to setup the most efficient, and cost
effictive platform, for hosting our cloud application
[Yoin-Vision](http://www.yoin-vision.com). Yoin-Vision offers facial analysis
as a service, for retail companies to gain insight in their visitors, perform
customer profiling, and offer extensive reporting, for measuring marketing
efficiency and store performance.

From a background as software engineers, we require a platform that is easy to
maintain, as well it should offer us the best performance. The platform will be
used for video analysis, and machine learning for data processing. We ended up
designing our platform, based on the micro-service architecture, using [The
Twelve-factor App](http://12factor.net/) methodology.

 > A instance should be designed and developed in a way, it can crash and be
 > relaunched, without compromising or affecting other instances in the
 > cluster.

Based on this methodology, we developed all our instances to run as Docker
containers. Giving us the freedom, in not being restricted to a single hosting
platform, but allowing us to run our containers on any platform that runs
Docker.

For testing and development, we used local and server based
[Fig](http://www.fig.sh/) deployments. We noticed that it would be cumbersome
to host Fig as a cluster, and orchestrating the cluster. We decided to go with
[Kubernetes](http://kubernetes.io) from Google, to be our docker cluster
orchestration platform. Allowing us to focus on developing Yoin-Vision.

For Yoin-Vision computing cost is an important aspect, as we are busy funding
our seed round. We have a [Microsoft
Bizspark](http://www.microsoft.com/bizspark) subscription, offering $750 of 
Microsoft Azure credits per month, divided to 5 accounts each with $150 per
month. Our goal was to maximize the use of this credit for our high-available
services (website, api, database). Second we use cheap dedicated hosting from
[Hetzner](http://www.hetzner.de/en), giving $50 machines with 8 cores, and
32GB RAM, that we will use for processing power.

# Requirements

The platform we will run, will be based on 4 Azure subscriptions. Per
subscription we launch 2 Azure Standard D1 instances, giving us most bang for
our buck. Next to the Azure instances, we will run 2+ dedicated servers, which
we can increase as we get more customers.

Kubernetes will be used as container orchestration platform. Ensuring our
containers are spawned throughout the cluster for improved redundancy. Second
it allows for defining where to host the containers, we define Azure as
high-available and Hetzner as dedicated.

Networking is setup using Weave Net, providing a easy to setup and maintain
network. We setup a container network, and a bridge network between hosts in
the cluster.

We will run our cluster on [CoreOS](http://www.coreos.com), as CoreOS is
designed for security, consistency, and reliability. Instead of
installing packages via yum or apt, CoreOS uses Linux containers to manage your
services at a higher level of abstraction. A single service’s code and all
dependencies are packaged within a container that can be run on one or many
CoreOS machines.

The following diagram displays our proposed cloud setup.
![Cloud
Architecture](https://pieterjong.files.wordpress.com/2015/07/weave-network-new-page1.png)


# Setup

For deploying our cluster, we have developed some deployment scripts. These
scripts enables us to deploy our cluster more efficient. The scripts are written
in Node.JS. The source of this deployment is hosted in my [Github
repository](https://github.com/jongpieter/coreos-guide/blob/master/coreos-weave-kubernetes/).
The source is based on the guide made by
[<NAME> (***@errordeveloper***)](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides/coreos/azure), on deploying Kubernetes and Weave to Coreos on Azure. 

The deployment steps are desribed in more detail in the following blog posts:
 - [Azure Cluster Setup](https://pieterjong.wordpress.com/2015/07/28/azure-cluster-setup/)
 - [Azure Local SSD optimisation](https://pieterjong.wordpress.com/2015/07/28/azure-local-ssd-optimization/)
 - [CoreOS Setup](https://pieterjong.wordpress.com/2015/07/27/coreos-cluster-setup/)
 - [Kubernetes Setup](https://pieterjong.wordpress.com/2015/07/27/kubernetes-cluster-setup/)
 - [Weave setup](https://pieterjong.wordpress.com/2015/07/27/weave-cluster-setup/)


## Azure

In setting up the Azure environment, we use the 5 accounts from our
[Microsoft Bizspark](http://www.microsoft.com/bizspark) subscription.
Requesting a Bizspark account can take up to 2 weeks, we received ours in 3
days after contact with the program manager of Bizspark Netherlands. You need
to allocate all accounts for maximum benefits.

Next you need to login to each Azure account, and setup a domain account, that
will be assigned `subscription administrator`. This account is used for the
command line deployment. Here is a great [guide on setting up this
account](https://azure.microsoft.com/en-us/documentation/articles/xplat-cli-connect/).
You need to do this for all accounts.
<commit_msg>Cosmetic<commit_after># Docker, CoreOS, Weave, Kubernetes Multi-Host/Cloud Cluster

Setting up and, running a Docker cluster was a completely new and unknown area
of expertise for us. Our goal was to setup the most efficient, and cost
effictive platform, for hosting our cloud application
[Yoin-Vision](http://www.yoin-vision.com). Yoin-Vision offers facial analysis
as a service, for retail companies to gain insight in their visitors, perform
customer profiling, and offer extensive reporting, for measuring marketing
efficiency and store performance.

From a background as software engineers, we require a platform that is easy to
maintain, as well it should offer us the best performance. The platform will be
used for video analysis, and machine learning for data processing. We ended up
designing our platform, based on the micro-service architecture, using [The
Twelve-factor App](http://12factor.net/) methodology.

 > A instance should be designed and developed in a way, it can crash and be
 > relaunched, without compromising or affecting other instances in the
 > cluster.

Based on this methodology, we developed all our instances to run as Docker
containers. Giving us the freedom, in not being restricted to a single hosting
platform, but allowing us to run our containers on any platform that runs
Docker.

For testing and development, we used local and server based
[Fig](http://www.fig.sh/) deployments. We noticed that it would be cumbersome
to host Fig as a cluster, and orchestrating the cluster. We decided to go with
[Kubernetes](http://kubernetes.io) from Google, to be our docker cluster
orchestration platform. Allowing us to focus on developing Yoin-Vision.

For Yoin-Vision computing cost is an important aspect, as we are busy funding
our seed round. We have a [Microsoft
Bizspark](http://www.microsoft.com/bizspark) subscription, offering $750 of 
Microsoft Azure credits per month, divided to 5 accounts each with $150 per
month. Our goal was to maximize the use of this credit for our high-available
services (website, api, database). Second we use cheap dedicated hosting from
[Hetzner](http://www.hetzner.de/en), giving $50 machines with 8 cores, and
32GB RAM, that we will use for processing power.

# Requirements

The platform we will run, will be based on 4 Azure subscriptions. Per
subscription we launch 2 Azure Standard D1 instances, giving us most bang for
our buck. Next to the Azure instances, we will run 2+ dedicated servers, which
we can increase as we get more customers.

Kubernetes will be used as container orchestration platform. Ensuring our
containers are spawned throughout the cluster for improved redundancy. Second
it allows for defining where to host the containers, we define Azure as
high-available and Hetzner as dedicated.

Networking is setup using Weave Net, providing a easy to setup and maintain
network. We setup a container network, and a bridge network between hosts in
the cluster.

We will run our cluster on [CoreOS](http://www.coreos.com), as CoreOS is
designed for security, consistency, and reliability. Instead of
installing packages via yum or apt, CoreOS uses Linux containers to manage your
services at a higher level of abstraction. A single service’s code and all
dependencies are packaged within a container that can be run on one or many
CoreOS machines.

The following diagram displays our proposed cloud setup.
![Cloud
Architecture](https://pieterjong.files.wordpress.com/2015/07/weave-network-new-page1.png)


# Setup

For deploying our cluster, we have developed some deployment scripts. These
scripts enables us to deploy our cluster more efficient. The scripts are written
in Node.JS. The source of this deployment is hosted in my [Github
repository](https://github.com/jongpieter/coreos-guide/blob/master/coreos-weave-kubernetes/).
The source is based on the guide made by
[<NAME> (**@errordeveloper**)](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides/coreos/azure), on deploying Kubernetes and Weave to Coreos on Azure. 

The deployment steps are desribed in more detail in the following blog posts:
 - [Azure Cluster Setup](https://pieterjong.wordpress.com/2015/07/28/azure-cluster-setup/)
 - [Azure Local SSD optimisation](https://pieterjong.wordpress.com/2015/07/28/azure-local-ssd-optimization/)
 - [CoreOS Setup](https://pieterjong.wordpress.com/2015/07/27/coreos-cluster-setup/)
 - [Kubernetes Setup](https://pieterjong.wordpress.com/2015/07/27/kubernetes-cluster-setup/)
 - [Weave setup](https://pieterjong.wordpress.com/2015/07/27/weave-cluster-setup/)


## Azure

In setting up the Azure environment, we use the 5 accounts from our
[Microsoft Bizspark](http://www.microsoft.com/bizspark) subscription.
Requesting a Bizspark account can take up to 2 weeks, we received ours in 3
days after contact with the program manager of Bizspark Netherlands. You need
to allocate all accounts for maximum benefits.

Next you need to login to each Azure account, and setup a domain account, that
will be assigned `subscription administrator`. This account is used for the
command line deployment. Here is a great [guide on setting up this
account](https://azure.microsoft.com/en-us/documentation/articles/xplat-cli-connect/).
You need to do this for all accounts.

{'fraction_non_alphanumeric': 0.06852288815478999, 'fraction_numerical': 0.012836243511090136, 'mean_word_length': 4.6998386229155455, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 11, 'https://': 18, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '820147', 'n_tokens_mistral': 2910, 'n_tokens_neox': 2678, 'n_words': 1380}