<commit_before>#Diffusion Benchmark Suite
##Overview
This project is a collection of synthetic benchmarks, or experiments, used to
drive performance of the Diffusion product. Our framework consists of:

* Ant scripts used to start/stop the Diffusion server (local/remote)
* And scripts for packaging and deploying publishers (by copying .dar files into
the deploy folder of the server)  

Currently implemented experiments are:

* Throughput experiment:<br>
    A broadcasting publisher is set up, publishing at a uniform rate across a 
    set number of topics. A population of clients subscribes to all topics. The
    experiment can be set up to examine server behaviour for different types of
    load resulting from a growing client population/increase of topics/increase
    of messages/message size. The experiment reports throughput from the clients
    perspective.
* Latency experiment:<br>
    A pong/echo publisher is set up which 'echo's clients messages back to them.
    The experiment allows for controlling the number concurrently pinging
    clients. Clients ping as fast as they can.
* Client churn test (coming soon):<br>
    A population of clients connects and subscribe, clients disconnect randomly
    after a set period of time.

##Building the benchmarks distributable
To build and run the benchmarks it is assumed that you have already installed:

* Ant (>1.8)
* Java JDK (>1.6)
* Diffusion (>4)

You will need to add the JDK and Ant bin folders to your PATH. <br>
You will also need to define DIFFUSION_HOME<br>
At this point you can use the build.xml in the project directory to build the
benchmarks distributable. Invoking __ant__ or __ant dist__ should result
in a packaged __benchmark-server.zip__ file being created in the project dir.<br>

##Deploying the benchmarks
The target machine used for running the benchmarks requires the same
dependencies described above for building (JDK/Ant/Diffusion). It also requires
a valid Diffusion licence file with sufficient allowances to support your test.
Similarly you will need to configure your server to support the required number
of connections and allow incoming connections on the desired ports. See the
diffusion server configuration and tuning manual for detailed instructions.
To deploy the scripts you'll need to extract the __benchmark-server.zip__ into
a folder named __benchmark-server__.<br>
>    $ unzip benchmark-server.zip -d benchmark-server

##Before you run the benchmarks!!!
The benchmarks are configured for running on tuned commodity servers, not your
average laptop. This is particularly true when running both server and client
on the same host. To run the benchmarks in your IDE refer to the relevant section
below.<br>
The server on which the benchmarks are to be run should be configured to allow
large numbers of connections. On Unix add file descriptors as follows:<br>
>    ulimit -n 100000

To persist the changes after a restart edit /etc/security/limits.conf (or /etc/limits.conf) and add the following lines:
>     *                hard    nofile          100000
>     *                soft    nofile          100000

If you run you client process via a single IP you may exhaust the IP range (by
opening too many local connections, each client requires a port). To
avoid you can either access your server via multiple addresses(explained below)
or extend the IP range as described [here](http://stackoverflow.com/questions/6145108/problem-running-into-java-net-bindexception-cannot-assign-requested-address)
  
##Running the benchmarks
The ant scripts deployed with the application support running the benchmarks
in a variety of configurations. The benchmarks can be run as a suite (similar
to junit test suites) with before/after tasks run before/after every benchmark.
Targets names which start with perfTest are assumed benchmarks. We'll work
through the options by example:

####Run the throughput suite against localhost.
>    $ ant -f throughput-suite.xml<br>

The default target for the throughput-suite.xml is the __perf-suite__ target
which will launch all the tests included. This run will default to testing the
Web Sockets transport with no conflation enabled.<br>
By the end of the run we will expect to have a set of experiment settings and
output files. The settings files are __not over written__ and thus allow for
manual tweaking. If you change the settings and re-run the settings from the
file will be used.

####Run a subset of the throughput suite against localhost.
>    $ ant -f throughput-suite.xml -Dtest.name.contains=125b<br>

This will result in only the __perfTest-125b-50t__ benchmark being run, but
within the framework of a suite so the before and after tasks are executed.

####Run the throughput suite against localhost with different transport/conflation
>    $ ant -f throughput-suite.xml -Ddiffusion.transport=dpt -Dconflation.mode=REPLACE<br>

This will result in a full suite run for the DPT transport using REPLACE conflation.
You can read more about the different Diffusion transports and conflation modes
in the User Manual.

####Run the throughput suite against localhost with different client jvm settings
>   $ ant -f throughput-suite.xml -Dclient.threads=1 -Dclient.jvm.args="-server -Xms128m -Xmx128m"

This will launch the full suite but with client jvm launched with above args
and using a single incoming thread.

####Run the throughput suite against a different machine
>    $ ant -f throughput-test.xml -Ddiffusion.host=test3<br>

If __diffusion.host__ is not localhost the ant scripts will attempt to
start/stop the diffusion server deployed remotely and deploy to it. The
assumption being made that the remote host is setup for running the benchmarks
in the same way the local machine is. The scripts use SSH to launch themselves
on the remote machine.<br>
This sucks a bit and we shall be looking to improve on this mechanism
shortly. For now however you will need to set the __ssh.username__ and
__ssh.username__ properties to allow the scripts ssh access. You'll also need
to have the jsch.jar in your $ANT_HOME/lib.

####Run the throughput benchmark with host on machine 'test3' connecting via 4 interfaces on the 192.168.54/24 and 10.0.0/24 networks.

>    $ ant -f throughput-test.xml -Ddiffusion.host=test3 -Ddiffusion.url=ws://192.168.54.31:8080,ws://10.0.0.10:8080,ws://10.0.0.18:8080,ws://10.0.0.22:8080 -Ddiffusion.client.nics=192.168.54.21,10.0.0.9,10.0.0.17,10.0.0.21 <br>

This will launch the full suite running the host on test3 and using several
urls and client NICs for connectivity. This allows you to mix and match
protocols and test your server using several outgoing/incoming network
interfaces. Note that the list of urls is used in conjunction with the list of
NICs and thus url[i] must be available on nic[i].

##Running/controlling tests manually


##Notes
### Notes for running in Eclipse
### Notes for IBM JDK
### Notes for Zing<commit_msg>more details, a link<commit_after>#Diffusion Benchmark Suite
##Overview
This project is a collection of synthetic benchmarks, or experiments, used to
drive performance of the [Diffusion](http://docs.pushtechnology.com/) product.
 Our framework consists of:

* Ant scripts used to start/stop the Diffusion server (local/remote)
* And scripts for packaging and deploying publishers (by copying .dar files into
the deploy folder of the server)
* Soft cushions!  

Currently implemented experiments are:

* Throughput experiment:<br>
    A broadcasting publisher is set up, publishing at a uniform rate across a 
    set number of topics. A population of clients subscribes to all topics. The
    experiment can be set up to examine server behaviour for different types of
    load resulting from a growing client population/increase of topics/increase
    of messages/message size. The experiment reports throughput from the clients
    perspective.
* Latency experiment:<br>
    A pong/echo publisher is set up which 'echo's clients messages back to them.
    The experiment allows for controlling the number concurrently pinging
    clients. Clients ping as fast as they can. The ping service can either
    send back to the particular client or broadcast on the ping topic.
* Remote Control Latency experiment:<br>
    A RemoteControl client is connected and sets up an Echo topic as described
    above. We connect clients to it and measure response time(Client <-> Server 
    <-> RC).
* Remote Control Throughput+Latency experiment:<br>
    A RemoteControl client is connected and sets up a topic tree similar to the
    one used in the throughput experiment. Latency is measured from
    RemoteControl client to clients (RC -> Server -> Client)

##Building the benchmarks distributable
To build and run the benchmarks it is assumed that you have already installed:

* Ant (>1.8)
* Java JDK (>1.6)
* Diffusion (>4)

You will need to add the JDK and Ant bin folders to your PATH. <br>
You will also need to define DIFFUSION_HOME<br>
At this point you can use the build.xml in the project directory to build the
benchmarks distributable. Invoking __ant__ or __ant dist__ should result
in a packaged __benchmark-server.zip__ file being created in the project dir.<br>

##Deploying the benchmarks
The target machine used for running the benchmarks requires the same
dependencies described above for building (JDK/Ant/Diffusion). It also requires
a valid Diffusion licence file with sufficient allowances to support your test.
Similarly you will need to configure your server to support the required number
of connections and allow incoming connections on the desired ports. See the
diffusion server configuration and tuning manual for detailed instructions.
To deploy the scripts you'll need to extract the __benchmark-server.zip__ into
a folder named __benchmark-server__.<br>
>    $ unzip benchmark-server.zip -d benchmark-server

##Before you run the benchmarks!!!
The benchmarks are configured for running on tuned commodity servers, not your
average laptop. This is particularly true when running both server and client
on the same host. To run the benchmarks in your IDE refer to the relevant section
below.<br>
The server on which the benchmarks are to be run should be configured to allow
large numbers of connections. On Unix add file descriptors as follows:<br>
>    ulimit -n 100000

To persist the changes after a restart edit /etc/security/limits.conf (or /etc/limits.conf) and add the following lines:
>     *                hard    nofile          100000
>     *                soft    nofile          100000

If you run you client process via a single IP you may exhaust the IP range (by
opening too many local connections, each client requires a port). To
avoid you can either access your server via multiple addresses(explained below)
or extend the IP range as described [here](http://stackoverflow.com/questions/6145108/problem-running-into-java-net-bindexception-cannot-assign-requested-address)
  
##Running the benchmarks
The ant scripts deployed with the application support running the benchmarks
in a variety of configurations. The benchmarks can be run as a suite (similar
to junit test suites) with before/after tasks run before/after every benchmark.
Targets names which start with perfTest are assumed benchmarks. We'll work
through the options by example:

####Run the throughput suite against localhost.
>    $ ant -f throughput-suite.xml<br>

The default target for the throughput-suite.xml is the __perf-suite__ target
which will launch all the tests included. This run will default to testing the
Web Sockets transport with no conflation enabled.<br>
By the end of the run we will expect to have a set of experiment settings and
output files. The settings files are __not over written__ and thus allow for
manual tweaking. If you change the settings and re-run the settings from the
file will be used.

####Run a subset of the throughput suite against localhost.
>    $ ant -f throughput-suite.xml -Dtest.name.contains=125b<br>

This will result in only the __perfTest-125b-50t__ benchmark being run, but
within the framework of a suite so the before and after tasks are executed.

####Run the throughput suite against localhost with different transport/conflation
>    $ ant -f throughput-suite.xml -Ddiffusion.transport=dpt -Dconflation.mode=REPLACE<br>

This will result in a full suite run for the DPT transport using REPLACE conflation.
You can read more about the different Diffusion transports and conflation modes
in the User Manual.

####Run the throughput suite against localhost with different client jvm settings
>   $ ant -f throughput-suite.xml -Dclient.threads=1 -Dclient.jvm.args="-server -Xms128m -Xmx128m"

This will launch the full suite but with client jvm launched with above args
and using a single incoming thread.

####Run the throughput suite against a different machine
>    $ ant -f throughput-test.xml -Ddiffusion.host=test3<br>

If __diffusion.host__ is not localhost the ant scripts will attempt to
start/stop the diffusion server deployed remotely and deploy to it. The
assumption being made that the remote host is setup for running the benchmarks
in the same way the local machine is. The scripts use SSH to launch themselves
on the remote machine.<br>
This sucks a bit and we shall be looking to improve on this mechanism
shortly. For now however you will need to set the __ssh.username__ and
__ssh.username__ properties to allow the scripts ssh access. You'll also need
to have the jsch.jar in your $ANT_HOME/lib.

####Run the throughput benchmark with host on machine 'test3' connecting via 4 interfaces on the 192.168.54/24 and 10.0.0/24 networks.

>    $ ant -f throughput-test.xml -Ddiffusion.host=test3 -Ddiffusion.url=ws://192.168.54.31:8080,ws://10.0.0.10:8080,ws://10.0.0.18:8080,ws://10.0.0.22:8080 -Ddiffusion.client.nics=192.168.54.21,10.0.0.9,10.0.0.17,10.0.0.21 <br>

This will launch the full suite running the host on test3 and using several
urls and client NICs for connectivity. This allows you to mix and match
protocols and test your server using several outgoing/incoming network
interfaces. Note that the list of urls is used in conjunction with the list of
NICs and thus url[i] must be available on nic[i].

##Running/controlling tests manually


##Notes
### Notes for running in Eclipse
If you have the Diffusion server installed locally on your developer box and
wish to run these benchmarks from Eclipse you can install the Diffusion Eclipse
Plugin and do just that! use the RunPublishers.launch(right click, run as)
launcher to start the service and create a launcher/use an existing one for your
selected experiment.<br>
Note that this is intended mostly as a means to validate your code, not as a
benchmarking environment for best results. 
### Notes for IBM JDK
### Notes for Zing
{'fraction_non_alphanumeric': 0.05461787739670537, 'fraction_numerical': 0.018498514717796382, 'mean_word_length': 4.181182231549493, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 16}, 'pii_count': 16, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4453169', 'n_tokens_mistral': 4082, 'n_tokens_neox': 3709, 'n_words': 2058}