<commit_before>hazelcast-stabilizer
===========================

A tool for stress testing Hazelcast and Hazelcast based applications in clustered environments. This can be in a local
machine, but can also be in a cloud like EC2 or Google Compute Engine. The Stabilizer makes use of JClouds, so in theory
we can roll out in any cloud.

### Mail Group

Please join the mail group if you are interested in using or developing Hazelcast.

[http://groups.google.com/group/hazelcast](http://groups.google.com/group/hazelcast)

#### License

Hazelcast Stabilizer is available under the Apache 2 License.

#### Copyright

Copyright (c) 2008-2013, Hazelcast, Inc. All Rights Reserved.

Visit [www.hazelcast.com](http://www.hazelcast.com/) for more info.


#### General structure

* Test: the functionality you want to test, e.g. a map. I looks a bit like a junit test, but it doesn't use annotations
and has a bunch of methods that one can override.

* TestSuite: this is a property file that contains the name of the Test class and the properties you want to set on that
test class instance. In most cases a testsuite contains a single test class, but you can configure multiple tests within
a single testsuite.

* Failure: an indication that something has gone wrong. E.g. the Worker crashed with an OOME, an exception occurred
while doing a map.get or the result of some test didn't give the expected answer. Failures are picked up by the Agent
and send back to the coordinator.

* Worker: a JVM responsible for running a TestSuite.

* Agent: a JVM installed on a piece of hardware. Its main responsibility is spawning, monitoring and terminating workers.

* Coordinator: a JVM that can run anywhere, e.g. on your local machine. You configure it with a list of Agents ip addresses
and you send a command like "run this testsuite with 10 worker JVM's for 2 hours".

* Provisioner: responsible for spawning/terminating EC2 instances and to install Agents on remote machines. It can be used
in combination with EC2 (or any other cloud), but it can also be used in a static setup like a local machine or the
test cluster we have in Istanbul office.

#### Installation

The zip/tar.gz file containing the stabilizer artifacts can be downloaded here:

```
https://oss.sonatype.org/content/repositories/snapshots/com/hazelcast/stabilizer/hazelcast-stabilizer-dist/0.3-SNAPSHOT/
```

Download and unpack the tar.gz or zip file to e.g. the home directory.

add to ~/.bashr:

```
export STABILIZER_HOME=~/hazelcast-stabilizer-0.3-SNAPSHOT
PATH=$STABILIZER_HOME/bin:$PATH
```

Create your tests working directory, e.g.

```
mkdir ~/tests
```

Copy the STABILIZER_HOME/conf/stabilizer.properties to the tests directory. And make
the changes required. In case of EC2, you only need to set your ec2 identity/credential:

```
CLOUD_IDENTITY=~/ec2.identity
CLOUD_CREDENTIAL=~/ec2.credential
```

The simplest thing you can do is to make a file '~/ec2.identity' containing your access key. And another
file '~/ec2.credential' containing your secret key. The reason why you need to create files instead of just
setting the values in this the stabilizer.properties, is security: it is too easy to share your credentials
with the outside world and now you can safely add the stabilizer.properties file in your source repo or
share it with other people.

### Setup public/private-key

After you have set up stabilizer, make sure you have a id_rsa.pub in your ~/.ssh directory. The Provisioner will
terminate when it needs to access the cloud and sees that the public/private key are missing. If you don't have
a public/private key, you can generate one like this:

```
ssh-keygen -t rsa -C "<EMAIL>"
```

You can press enter on all questions. The value for the email address is not relevant. After this command has completed, you
should have a ida_rsa.pub and id_rsa file in your ~/.ssh directory. Your id_rsa.pub key will automatically be copied to
the remote agent machines and added to the ~/.ssh/known_hosts file, so that you can log into that machine without
a password or explicit provided credentials.

#### Set up on Google Compute Engine

For setup on GCE a developer email can be obtained from the GCE admin GUI console.
Its usually something in the form: <my account id>@<EMAIL>
go to API & Auth > Credentials, click Create New Client ID,  select Service Acount.
save your p12 keystore file that is obtained when creating a "Service Account"

run ./setupGce.sh script
provide you <my <EMAIL> as 1st argument
provide and the path to your p12 file as 2nd argument
the setupGce.sh script will create your stabilizer.properties, in the conf directory


### Using the archetype

Probably you want to write your own test. The easiest way to do that is to make use of the Stabilizer archetype which
will generate a project for you.

```
mvn archetype:generate -DarchetypeRepository=https://oss.sonatype.org/content/repositories/snapshots \
    -DarchetypeGroupId=com.hazelcast.stabilizer \
    -DarchetypeArtifactId=archetype \
    -DarchetypeVersion=0.3-SNAPSHOT \
    -DgroupId=yourgroupid  \
    -DartifactId=yourproject
```

After this project is generated, go to the created directory and run:

```
mvn clean install
```

And then go to workdir and edit the stabilizer.properties file. In case of EC2, you only need to alter the following:

```
CLOUD_IDENTITY=~/ec2.identity
CLOUD_CREDENTIAL=~/ec2.credential
```

The ec2.identity file should contain your access key and the ec2.credential your secret key.

After you have made the modifications, you need to set the executable flag on the run.sh:

```
chmod +x run.sh
```

This is needed because the maven archetype is not able to deal with the executable flag.

And finally you can run the test:<commit_msg>clarification of the run.sh location<commit_after>hazelcast-stabilizer
===========================

A tool for stress testing Hazelcast and Hazelcast based applications in clustered environments. This can be in a local
machine, but can also be in a cloud like EC2 or Google Compute Engine. The Stabilizer makes use of JClouds, so in theory
we can roll out in any cloud.

### Mail Group

Please join the mail group if you are interested in using or developing Hazelcast.

[http://groups.google.com/group/hazelcast](http://groups.google.com/group/hazelcast)

#### License

Hazelcast Stabilizer is available under the Apache 2 License.

#### Copyright

Copyright (c) 2008-2013, Hazelcast, Inc. All Rights Reserved.

Visit [www.hazelcast.com](http://www.hazelcast.com/) for more info.


#### General structure

* Test: the functionality you want to test, e.g. a map. I looks a bit like a junit test, but it doesn't use annotations
and has a bunch of methods that one can override.

* TestSuite: this is a property file that contains the name of the Test class and the properties you want to set on that
test class instance. In most cases a testsuite contains a single test class, but you can configure multiple tests within
a single testsuite.

* Failure: an indication that something has gone wrong. E.g. the Worker crashed with an OOME, an exception occurred
while doing a map.get or the result of some test didn't give the expected answer. Failures are picked up by the Agent
and send back to the coordinator.

* Worker: a JVM responsible for running a TestSuite.

* Agent: a JVM installed on a piece of hardware. Its main responsibility is spawning, monitoring and terminating workers.

* Coordinator: a JVM that can run anywhere, e.g. on your local machine. You configure it with a list of Agents ip addresses
and you send a command like "run this testsuite with 10 worker JVM's for 2 hours".

* Provisioner: responsible for spawning/terminating EC2 instances and to install Agents on remote machines. It can be used
in combination with EC2 (or any other cloud), but it can also be used in a static setup like a local machine or the
test cluster we have in Istanbul office.

#### Installation

The zip/tar.gz file containing the stabilizer artifacts can be downloaded here:

```
https://oss.sonatype.org/content/repositories/snapshots/com/hazelcast/stabilizer/hazelcast-stabilizer-dist/0.3-SNAPSHOT/
```

Download and unpack the tar.gz or zip file to e.g. the home directory.

add to ~/.bashr:

```
export STABILIZER_HOME=~/hazelcast-stabilizer-0.3-SNAPSHOT
PATH=$STABILIZER_HOME/bin:$PATH
```

Create your tests working directory, e.g.

```
mkdir ~/tests
```

Copy the STABILIZER_HOME/conf/stabilizer.properties to the tests directory. And make
the changes required. In case of EC2, you only need to set your ec2 identity/credential:

```
CLOUD_IDENTITY=~/ec2.identity
CLOUD_CREDENTIAL=~/ec2.credential
```

The simplest thing you can do is to make a file '~/ec2.identity' containing your access key. And another
file '~/ec2.credential' containing your secret key. The reason why you need to create files instead of just
setting the values in this the stabilizer.properties, is security: it is too easy to share your credentials
with the outside world and now you can safely add the stabilizer.properties file in your source repo or
share it with other people.

### Setup public/private-key

After you have set up stabilizer, make sure you have a id_rsa.pub in your ~/.ssh directory. The Provisioner will
terminate when it needs to access the cloud and sees that the public/private key are missing. If you don't have
a public/private key, you can generate one like this:

```
ssh-keygen -t rsa -C "<EMAIL>"
```

You can press enter on all questions. The value for the email address is not relevant. After this command has completed, you
should have a ida_rsa.pub and id_rsa file in your ~/.ssh directory. Your id_rsa.pub key will automatically be copied to
the remote agent machines and added to the ~/.ssh/known_hosts file, so that you can log into that machine without
a password or explicit provided credentials.

#### Set up on Google Compute Engine

For setup on GCE a developer email can be obtained from the GCE admin GUI console.
Its usually something in the form: <my account id>@<EMAIL>
go to API & Auth > Credentials, click Create New Client ID,  select Service Acount.
save your p12 keystore file that is obtained when creating a "Service Account"

run ./setupGce.sh script
provide you <my account id>@<EMAIL> as 1st argument
provide and the path to your p12 file as 2nd argument
the setupGce.sh script will create your stabilizer.properties, in the conf directory


### Using the archetype

Probably you want to write your own test. The easiest way to do that is to make use of the Stabilizer archetype which
will generate a project for you.

```
mvn archetype:generate -DarchetypeRepository=https://oss.sonatype.org/content/repositories/snapshots \
    -DarchetypeGroupId=com.hazelcast.stabilizer \
    -DarchetypeArtifactId=archetype \
    -DarchetypeVersion=0.3-SNAPSHOT \
    -DgroupId=yourgroupid  \
    -DartifactId=yourproject
```

After this project is generated, go to the created directory and run:

```
mvn clean install
```

And then go to workdir and edit the stabilizer.properties file. In case of EC2, you only need to alter the following:

```
CLOUD_IDENTITY=~/ec2.identity
CLOUD_CREDENTIAL=~/ec2.credential
```

The ec2.identity file should contain your access key and the ec2.credential your secret key.

After you have made the modifications, you need to set the executable flag on the run.sh in the workdir directory:

```
chmod +x run.sh
```

This is needed because the maven archetype is not able to deal with the executable flag.

And finally you can run the test:
{'fraction_non_alphanumeric': 0.07007969507969508, 'fraction_numerical': 0.006583506583506584, 'mean_word_length': 4.240581025873809, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6283760', 'n_tokens_mistral': 3387, 'n_tokens_neox': 3213, 'n_words': 1638}