<commit_before>Spark Analytics - Build your own Experiment
-------------------------------------------

#### Data-Source
All the collected data of your _metered_ simulators/tasks is available in **InfluxDB** via the [InfluxDB-API](http://influxdb.com/docs/v0.7/api/reading_and_writing_data.html).

We put together a [simple library](https://github.com/att-innovate/charmander-spark/blob/master/src/main/scala/org/att/charmander/CharmanderUtils.scala)
that gives you easy access to the data from Spark/Scala. Some available functions are:

    def getMeteredTaskNamesFromRedis(): List[String]
    def getRDDForTask(sc: SparkContext, taskName: String, attributeName: String, numberOfPoints: Int): RDD[List[BigDecimal]]
    def getRDDForNode(sc: SparkContext, nodeName: String, attributeName: String, numberOfPoints: Int): RDD[List[BigDecimal]]

Example: Retrieve the currently active metered tasks:

    val tasks = CharmanderUtils.getMeteredTaskNamesFromRedis

Example: Get 200 memory_usage data-points for lookbusy200 simulator:

    val memoryUsage= CharmanderUtils.getRDDForTask(sc, "lookbusy200", "memory_usage", 200)

Example: Get 200 memory_usage data-points for the _slave2_ node:

    val memoryUsage= CharmanderUtils.getRDDForNode(sc, "slave2", "memory_usage", 200)


#### Destination for Task-Intelligence
The output of your analysis can be fed back in to the scheduler via Redis. The keys in Redis have to be in a certain form
to be understood by the Charmander-Scheduler.

To simplify the setting of those findings our [library](https://github.com/att-innovate/charmander-spark/blob/master/src/main/scala/org/att/charmander/CharmanderUtils.scala)
offers following function:

    def setTaskIntelligence(taskName: String, attributeName: String, value: String)
    def getTaskIntelligence(taskName: String, attributeName: String): String

Example: Overwrite the memory allocation for lookbusy200 to 400

    // Redis key to overwrite memory allocation: charmander:task-intelligence:lookbusy200:mem
    CharmanderUtils.setTaskIntelligence("lookbusy200", "mem", "400")

Example: Retrieve previously set memory allocation for lookbusy200

    val mem = CharmanderUtils.getTaskIntelligence("lookbusy200", "mem")

Example: Force lookbusy200 to always be deployed to slave2

    // Redis key to overwrite memory allocation: charmander:task-intelligence:lookbusy200:nodename
    CharmanderUtils.setTaskIntelligence("lookbusy200", "nodename", "slave2")


#### Build and use our Spark utilties library locally
Use the library that comes with the Charmander project. Copy that library in to your `/tmp` directory for example
and use it with Spark-shell:

    git clone https://github.com/att-innovate/charmander
    cp charmander/analytics/spark/sparkkernel/files/charmander-utils_2.10-1.0.jar /tmp

    //cd to your local spark deployment
    cd spark-1.2.0
    ./bin/spark-shell --jars /tmp/charmander-utils_2.10-1.0.jar

    //verify that you can import the library
    import org.att.charmander.CharmanderUtils

And of course you can build the library yourself:

    git clone https://github.com/att-innovate/charmander-spark
    cd charmander-spark

    // verify that you have Java and sbt (>= 0.13.6) installed
    java -version
    sbt --version

    // build it
    sbt assemble

    // copy the library to the tmp directoy
    cp target/scala-2.10/charmander-utils_2.10-1.0.jar /tmp
<commit_msg>Fix - documentation<commit_after>Spark Analytics - Build your own Experiment
-------------------------------------------

#### Data-Source
All the collected data of your _metered_ simulators/tasks is available in **InfluxDB** via the [InfluxDB-API](http://influxdb.com/docs/v0.7/api/reading_and_writing_data.html).

We put together a [simple library](https://github.com/att-innovate/charmander-spark/blob/master/src/main/scala/org/att/charmander/CharmanderUtils.scala)
that gives you easy access to the data from Spark/Scala. Some available functions are:

    def getMeteredTaskNamesFromRedis(): List[String]
    def getRDDForTask(sc: SparkContext, taskName: String, attributeName: String, numberOfPoints: Int): RDD[List[BigDecimal]]
    def getRDDForNode(sc: SparkContext, nodeName: String, attributeName: String, numberOfPoints: Int): RDD[List[BigDecimal]]

Example: Retrieve the currently active metered tasks:

    val tasks = CharmanderUtils.getMeteredTaskNamesFromRedis

Example: Get 200 memory_usage data-points for lookbusy200 simulator:

    val memoryUsage= CharmanderUtils.getRDDForTask(sc, "lookbusy200", "memory_usage", 200)

Example: Get 200 memory_usage data-points for the _slave2_ node:

    val memoryUsage= CharmanderUtils.getRDDForNode(sc, "slave2", "memory_usage", 200)


#### Destination for Task-Intelligence
The output of your analysis can be fed back in to the scheduler via Redis. The keys in Redis have to be in a certain form
to be understood by the Charmander-Scheduler.

To simplify the setting of those findings our [library](https://github.com/att-innovate/charmander-spark/blob/master/src/main/scala/org/att/charmander/CharmanderUtils.scala)
offers following function:

    def setTaskIntelligence(taskName: String, attributeName: String, value: String)
    def getTaskIntelligence(taskName: String, attributeName: String): String

Example: Overwrite the memory allocation for lookbusy200 to 400

    // Redis key to overwrite memory allocation: charmander:task-intelligence:lookbusy200:mem
    CharmanderUtils.setTaskIntelligence("lookbusy200", "mem", "400")

Example: Retrieve previously set memory allocation for lookbusy200

    val mem = CharmanderUtils.getTaskIntelligence("lookbusy200", "mem")

Example: Force lookbusy200 to always be deployed to slave2

    // Redis key to overwrite memory allocation: charmander:task-intelligence:lookbusy200:nodename
    CharmanderUtils.setTaskIntelligence("lookbusy200", "nodename", "slave2")


#### Build and use our Spark utilties library locally
Use the library that comes with the Charmander project. Copy that library in to your `/tmp` directory for example
and use it with Spark-shell:

    git clone https://github.com/att-innovate/charmander
    cp charmander/analytics/spark/sparkkernel/files/charmander-utils_2.10-1.0.jar /tmp

    //cd to your local spark deployment
    cd spark-1.2.0
    ./bin/spark-shell --jars /tmp/charmander-utils_2.10-1.0.jar

    //verify that you can import the library
    import org.att.charmander.CharmanderUtils

And of course you can build the library yourself:

    git clone https://github.com/att-innovate/charmander-spark
    cd charmander-spark

    // verify that you have Java and sbt (>= 0.13.6) installed
    java -version
    sbt --version

    // build it
    sbt package

    // copy the library to the tmp directoy
    cp target/scala-2.10/charmander-utils_2.10-1.0.jar /tmp

{'fraction_non_alphanumeric': 0.10024952297079114, 'fraction_numerical': 0.023190958461764274, 'mean_word_length': 4.1935975609756095, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6163188', 'n_tokens_mistral': 2182, 'n_tokens_neox': 1998, 'n_words': 627}