<commit_before>Orka
=====

###Overview

orka is a command-line tool, and also a client development library, for creating and deleting Hadoop (YARN) clusters of virtual machines in ~okeanos IaaS.

##Setup user environment to run orka (Debian)

    sudo apt-get update
    sudo apt-get install -y git
    sudo apt-get install -y python python-dev python-pip

##Adding [orka]      section to kamaki config file

User should open ~/.kamakirc and append these two lines:
    
    [orka]                                                              
    base_url = < e-science -IP- or -url address- >

###Virtual environment

Optional but highly recommended is to install and use the orka package in a virtual environment:
 
    sudo pip install virtualenv
    mkdir .virtualenvs
    cd .virtualenvs
    virtualenv --system-site-packages orkaenv
    . ../.virtualenvs/orkaenv/bin/activate
    (with deactivate from command line you exit the virtual env)
    
Following commands download and install orka (either directly or in virtual environment):

    [sudo if not using virtualenv] pip install ansible==1.7.2
    cd
    git clone <escience git repo> <project_name> 
    cd to <project_name>/orka
    [sudo if not using virtualenv] python setup.py install
 
  Now orka commands are usable from anywhere.

##How to run orka commands

$ orka [command] "arguments"

Optional arguments for all orka commands:
    
    --auth_url="authentication url (default value='https://accounts.okeanos.grnet.gr/identity/v2.0')",
    --token="an ~okeanos token (default value read from ~/.kamakirc)",
    --server_url="orka web application url (default value read from ~/.kamakirc)",

## "images" command

Images command has no required positional or optional arguments.

### {orka images} command example

example for listing available images and their pithos uuid values using defaults from .kamakirc

    orka images 

## "create" command

Required positional arguments for create command:
         
    name: "name of the cluster" 
    cluster_size: "total VMs, including master node" 
    cpu_master: "master node: number of CPU cores" 
    ram_master: "master node: ram in MB",
    disk_master: "master node: hard drive in GB",
    cpu_slave: "each slave node: number of CPU cores",
    ram_slave: "each slave node: ram in MB",
    disk_slave: "each slave node: hard drive in GB",
    disk_template: "Standard or Archipelago"
    project_name: "name of a ~okeanos project, to pull resources from"
    
Optional arguments for create command:

    --image="Operating System (default value='Debian Base')",
(available images can be found with **orka images** command)
    --replication_factor="HDFS replication factor. Default is 2",
    --dfs_blocksize="HDFS block size (in MB). Default is 128",

### Create Hadoop cluster from a pre-configured image

Using the --image=Hadoop-2.5.2-Debian-8.0 argument creates the Hadoop cluster much faster because it utilises a specially created ~okeanos VM image with Java and YARN pre-installed. Omitting this argument ensures that the latest stable YARN version will be installed (but at the cost of lower speed).

###{orka create} command examples

example for create cluster with default optionals (not hadoop_image):

    orka create Yarn_Test 2 2 2048 10 2 1024 10 Archipelago <project_name>
<commit_msg>Fix leftover reference to old image name in readme<commit_after>Orka
=====

###Overview

orka is a command-line tool, and also a client development library, for creating and deleting Hadoop (YARN) clusters of virtual machines in ~okeanos IaaS.

##Setup user environment to run orka (Debian)

    sudo apt-get update
    sudo apt-get install -y git
    sudo apt-get install -y python python-dev python-pip

##Adding [orka]      section to kamaki config file

User should open ~/.kamakirc and append these two lines:
    
    [orka]                                                              
    base_url = < e-science -IP- or -url address- >

###Virtual environment

Optional but highly recommended is to install and use the orka package in a virtual environment:
 
    sudo pip install virtualenv
    mkdir .virtualenvs
    cd .virtualenvs
    virtualenv --system-site-packages orkaenv
    . ../.virtualenvs/orkaenv/bin/activate
    (with deactivate from command line you exit the virtual env)
    
Following commands download and install orka (either directly or in virtual environment):

    [sudo if not using virtualenv] pip install ansible==1.7.2
    cd
    git clone <escience git repo> <project_name> 
    cd to <project_name>/orka
    [sudo if not using virtualenv] python setup.py install
 
  Now orka commands are usable from anywhere.

##How to run orka commands

$ orka [command] "arguments"

Optional arguments for all orka commands:
    
    --auth_url="authentication url (default value='https://accounts.okeanos.grnet.gr/identity/v2.0')",
    --token="an ~<PASSWORD> (default value read from ~/.kamakirc)",
    --server_url="orka web application url (default value read from ~/.kamakirc)",

## "images" command

Images command has no required positional or optional arguments.

### {orka images} command example

example for listing available images and their pithos uuid values using defaults from .kamakirc

    orka images 

## "create" command

Required positional arguments for create command:
         
    name: "name of the cluster" 
    cluster_size: "total VMs, including master node" 
    cpu_master: "master node: number of CPU cores" 
    ram_master: "master node: ram in MB",
    disk_master: "master node: hard drive in GB",
    cpu_slave: "each slave node: number of CPU cores",
    ram_slave: "each slave node: ram in MB",
    disk_slave: "each slave node: hard drive in GB",
    disk_template: "Standard or Archipelago"
    project_name: "name of a ~okeanos project, to pull resources from"
    
Optional arguments for create command:

    --image="Operating System (default value='Debian Base')",
(available images can be found with **orka images** command)
    --replication_factor="HDFS replication factor. Default is 2",
    --dfs_blocksize="HDFS block size (in MB). Default is 128",

### Create Hadoop cluster from a pre-configured image

Using the --image=Hadoop-2.5.2 argument creates the Hadoop cluster much faster because it utilises a specially created ~okeanos VM image with Java and YARN pre-installed. Omitting this argument ensures that the latest stable YARN version will be installed (but at the cost of lower speed).

###{orka create} command examples

example for create cluster with default optionals (not hadoop_image):

    orka create Yarn_Test 2 2 2048 10 2 1024 10 Archipelago <project_name>

{'fraction_non_alphanumeric': 0.0779590004489002, 'fraction_numerical': 0.008379470297770463, 'mean_word_length': 3.3772102161100195, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7079551', 'n_tokens_mistral': 1960, 'n_tokens_neox': 1812, 'n_words': 821}