<commit_before># General Overview
Stone Ridge is a combination of dedicated hardware and software to test
networking performance in Firefox under real-world(-ish) conditions. We
simulate three different kinds of networks (called "netconfigs").

1. Broadband similar to what those of us in the US have at home ("broadband")
2. Cellular traffic similar to what those of us in the US have ("umts")
3. Cellular traffic similar to what you might find in less developed parts of the world ("gsm")

We are able to test against each of these netconfigs on all of our Tier-1
desktop platforms (Mac OS X 10.8, Windows 7, and Linux (CentOS 6)).

The testbed is made up of 6 machines: 4 linux machines, 1 mac mini, and 1
windows machine. These machines are:

stone-ridge-linux1.dmz.scl3.mozilla.com - Stone Ridge Master and server for broadband netconfig
stone-ridge-linux2.dmz.scl3.mozilla.com - Server for umts netconfig
stone-ridge-linux3.dmz.scl3.mozilla.com - Server for gsm netconfig
stone-ridge-linux4.dmz.scl3.mozilla.com - Linux client machine
stone-ridge-mac1.dmz.scl3.mozilla.com - Mac OS X client machine
stone-ridge-win1.dmz.scl3.mozilla.com - Windows 7 client machine

None of these machines use any virtualization because we can't risk the sharing
of resources that comes with virtualization that could affect the test numbers.

# Machine Roles

## Master
stone-ridge-linux1.dmz.scl3.mozilla.com

This is the only machine that needs to talk to anything outside the stone
ridge testbed. Its main functions are

* Download builds from ftp.m.o
* Serve those builds to the clients
* Receive stone ridge push requests for "try"-like runs
* Schedule nightly stone ridge runs
* Tell other machines what tests to run and when
* Receive results from test runs and forward them on to datazilla.m.o
* Email results from "try"-like runs to the user that requested them

The queue of stone ridge runs that are waiting or being processed is stored in
a RabbitMQ instance running on the master.

### Special Windows Considerations
Since windows is a pain in the you-know-where, it is prone to losing its
connection to the RabbitMQ instance running on the master. To mitigate this,
I have written a web-based "mq" that is entirely for jobs that need to run on
the windows client. Instead of talking to RabbitMQ directly, the windows client
periodically asks the webmq if there are any jobs for it, and runs them if so.
There is a special process running on the master that takes messages destined
for windows out of RabbitMQ and places them into an sqlite database used by the
webmq, which then serves them to the windows client.

### Processes running on the master

#### rabbitmq
This is the message queue server that everything talks to to find out what it
needs to do

#### nginx
This is the server that is used to serve builds of firefox to the clients

#### srmaster
This is the process that takes requests to run stone ridge tests (either
the nightly cron job or a "pushed" test against a try build) and makes them
ready to run. It does this by running srcloner to do the downloading of builds.
Once the builds have been successfully downloaded, this process places messages
in each of the servers' queues to let them know that they need to schedule tests
to be run against their netconfig.

#### srcloner
These are processes that are used to download builds from ftp.m.o. If the
download succeeds, it exits cleanly so the master knows the build is ready
to be used for testing. Otherwise, it will exit uncleanly so the master knows
to throw away this attempt for the build. If it's been less than a day since the
first time we tried to download this build, it will kick off a deferrer process
(see below) before exiting.

#### srdeferrer
These are processes that sleep for a while (to give try builds a chance to
finish) and then re-insert them as "new" runs in the master.

#### srmqproxy
This is the process that takes messages bound for the windows client out of
RabbitMQ and puts them into a sqlite database.

#### srwebmq
This is the process that serves MQ messages to the windows client over a web
service.

#### srreporter
This is the process that reports test results from the clients to datazilla.m.o,
and emails successful test results from "pushed" runs to the user.

#### sremailer
This exposes a web service for machines that are not the master to send emails
outside the testbed (used by clients to send failed test notifications).

#### cron jobs
There are two cron jobs running on the master. One that runs every night at 5am
Pacific, to kick off a full run of tests on the latest nightly build, and one
that runs every minute to check the "push" server (currently hosted on heroku)
to see if anyone has asked to run tests against a try build.

## Server
stone-ridge-linux1.dmz.scl3.mozilla.com (broadband)
stone-ridge-linux2.dmz.scl3.mozilla.com (umts)
stone-ridge-linux3.dmz.scl3.mozilla.com (gsm)

These machines are responsible for shaping network traffic to and from them to
make it behave like the netconfig they are assigned to. Additionally, they serve
test content to the clients via HTTP, as well as acting as DNS servers for
the clients that are configured to be talking to them (so we can use real
hostnames but have their traffic directed over a properly simulated network).
They also record traffic during a test run to be used for debugging purposes in
case something goes wrong, or we wonder why a particular test behaves the way it
does.

### Processes running on the servers

#### apache
This is a special build of apache, with some modifications to mod\_rewrite that
allows for serving files with their original headers (as recorded from the
public internet) and allowing different versions of a URL based on the query
string. This is what serves test content to the clients.

#### srscheduler
This receives test requests from the master and farms them out to the clients as
appropriate. This is mostly an artifact from the time when I was intending to
only have one client sending traffic to a server at a time (to reduce
interference between clients). That plan was thrown out the window by windows
being flaky with RabbitMQ. Instead of removing this process entirely, I just
made it dumber to avoid having to rearchitect things even further. Fortunately,
interaction between multiple clients and one server seems to have little
(if any) effect on the test results.

#### srnamed
This serves DNS responses to the clients. For the most part, it only replies
the server's own IP address, but for a few limited hostnames (each machine in
the stone ridge testbed, as well as puppet) it responds with the actual IPs of
those machines.

#### srpcapper
This runs tcpdump for every test that is run against the server, and serves the
final pcap to the clients for archival. The reason we run this on the server
instead of on the clients is that I was unable to make OS X or Win7 reliably
give a full pcap of the traffic.

#### interface startup
A script that runs on system startup to configure network traffic shaping on the
interface that is connected to the stone ridge switch.

## Client
stone-ridge-linux4.dmz.scl3.mozilla.com (CentOS 6)
stone-ridge-mac1.dmz.scl3.mozilla.com (Mac OS X 10.8)
stone-ridge-win1.dmz.scl3.mozilla.com (Windows 7)

These machines run the tests against each of the selected netconfig servers in
turn, record the results, and report them to the master. Each of these machines
has its monitor port plugged into an Extron EDID and has a user automatically
logged in on the graphical terminal. This user has a terminal in its startup
items, and it is this terminal that runs the worker process. We do this because,
like TpWhateverWeCallItNow, we need a way to display an actual firefox for some
of the stone ridge tests.

### Processes running on the clients

#### srworker<commit_msg>Add notes on connectivity, fix some formatting<commit_after># General Overview
Stone Ridge is a combination of dedicated hardware and software to test
networking performance in Firefox under real-world(-ish) conditions. We
simulate three different kinds of networks (called "netconfigs").

1. Broadband similar to what those of us in the US have at home ("broadband")
2. Cellular traffic similar to what those of us in the US have ("umts")
3. Cellular traffic similar to what you might find in less developed parts of the world ("gsm")

We are able to test against each of these netconfigs on all of our Tier-1
desktop platforms (Mac OS X 10.8, Windows 7, and Linux (CentOS 6)).

The testbed is made up of 6 machines: 4 linux machines, 1 mac mini, and 1
windows machine. They are all accessible via the MPT VPN. These machines are:

* stone-ridge-linux1.dmz.scl3.mozilla.com - Stone Ridge Master and server for broadband netconfig
* stone-ridge-linux2.dmz.scl3.mozilla.com - Server for umts netconfig
* stone-ridge-linux3.dmz.scl3.mozilla.com - Server for gsm netconfig
* stone-ridge-linux4.dmz.scl3.mozilla.com - Linux client machine
* stone-ridge-mac1.dmz.scl3.mozilla.com - Mac OS X client machine
* stone-ridge-win1.dmz.scl3.mozilla.com - Windows 7 client machine NOTE: RDP runs on port 22 here instead of the default

None of these machines use any virtualization because we can't risk the sharing
of resources that comes with virtualization that could affect the test numbers.

# Machine Roles

## Master
stone-ridge-linux1.dmz.scl3.mozilla.com

This is the only machine that needs to talk to anything outside the stone
ridge testbed. Its main functions are

* Download builds from ftp.m.o
* Serve those builds to the clients
* Receive stone ridge push requests for "try"-like runs
* Schedule nightly stone ridge runs
* Tell other machines what tests to run and when
* Receive results from test runs and forward them on to datazilla.m.o
* Email results from "try"-like runs to the user that requested them

The queue of stone ridge runs that are waiting or being processed is stored in
a RabbitMQ instance running on the master.

### Special Windows Considerations
Since windows is a pain in the you-know-where, it is prone to losing its
connection to the RabbitMQ instance running on the master. To mitigate this,
I have written a web-based "mq" that is entirely for jobs that need to run on
the windows client. Instead of talking to RabbitMQ directly, the windows client
periodically asks the webmq if there are any jobs for it, and runs them if so.
There is a special process running on the master that takes messages destined
for windows out of RabbitMQ and places them into an sqlite database used by the
webmq, which then serves them to the windows client.

### Processes running on the master

#### rabbitmq
This is the message queue server that everything talks to to find out what it
needs to do

#### nginx
This is the server that is used to serve builds of firefox to the clients

#### srmaster
This is the process that takes requests to run stone ridge tests (either
the nightly cron job or a "pushed" test against a try build) and makes them
ready to run. It does this by running srcloner to do the downloading of builds.
Once the builds have been successfully downloaded, this process places messages
in each of the servers' queues to let them know that they need to schedule tests
to be run against their netconfig.

#### srcloner
These are processes that are used to download builds from ftp.m.o. If the
download succeeds, it exits cleanly so the master knows the build is ready
to be used for testing. Otherwise, it will exit uncleanly so the master knows
to throw away this attempt for the build. If it's been less than a day since the
first time we tried to download this build, it will kick off a deferrer process
(see below) before exiting.

#### srdeferrer
These are processes that sleep for a while (to give try builds a chance to
finish) and then re-insert them as "new" runs in the master.

#### srmqproxy
This is the process that takes messages bound for the windows client out of
RabbitMQ and puts them into a sqlite database.

#### srwebmq
This is the process that serves MQ messages to the windows client over a web
service.

#### srreporter
This is the process that reports test results from the clients to datazilla.m.o,
and emails successful test results from "pushed" runs to the user.

#### sremailer
This exposes a web service for machines that are not the master to send emails
outside the testbed (used by clients to send failed test notifications).

#### cron jobs
There are two cron jobs running on the master. One that runs every night at 5am
Pacific, to kick off a full run of tests on the latest nightly build, and one
that runs every minute to check the "push" server (currently hosted on heroku)
to see if anyone has asked to run tests against a try build.

## Server
* stone-ridge-linux1.dmz.scl3.mozilla.com (broadband)
* stone-ridge-linux2.dmz.scl3.mozilla.com (umts)
* stone-ridge-linux3.dmz.scl3.mozilla.com (gsm)

These machines are responsible for shaping network traffic to and from them to
make it behave like the netconfig they are assigned to. Additionally, they serve
test content to the clients via HTTP, as well as acting as DNS servers for
the clients that are configured to be talking to them (so we can use real
hostnames but have their traffic directed over a properly simulated network).
They also record traffic during a test run to be used for debugging purposes in
case something goes wrong, or we wonder why a particular test behaves the way it
does.

### Processes running on the servers

#### apache
This is a special build of apache, with some modifications to mod\_rewrite that
allows for serving files with their original headers (as recorded from the
public internet) and allowing different versions of a URL based on the query
string. This is what serves test content to the clients.

#### srscheduler
This receives test requests from the master and farms them out to the clients as
appropriate. This is mostly an artifact from the time when I was intending to
only have one client sending traffic to a server at a time (to reduce
interference between clients). That plan was thrown out the window by windows
being flaky with RabbitMQ. Instead of removing this process entirely, I just
made it dumber to avoid having to rearchitect things even further. Fortunately,
interaction between multiple clients and one server seems to have little
(if any) effect on the test results.

#### srnamed
This serves DNS responses to the clients. For the most part, it only replies
the server's own IP address, but for a few limited hostnames (each machine in
the stone ridge testbed, as well as puppet) it responds with the actual IPs of
those machines.

#### srpcapper
This runs tcpdump for every test that is run against the server, and serves the
final pcap to the clients for archival. The reason we run this on the server
instead of on the clients is that I was unable to make OS X or Win7 reliably
give a full pcap of the traffic.

#### interface startup
A script that runs on system startup to configure network traffic shaping on the
interface that is connected to the stone ridge switch.

## Client
* stone-ridge-linux4.dmz.scl3.mozilla.com (CentOS 6)
* stone-ridge-mac1.dmz.scl3.mozilla.com (Mac OS X 10.8)
* stone-ridge-win1.dmz.scl3.mozilla.com (Windows 7) NOTE: RDP runs on port 22 here instead of the default.

These machines run the tests against each of the selected netconfig servers in
turn, record the results, and report them to the master. Each of these machines
has its monitor port plugged into an Extron EDID and has a user automatically
logged in on the graphical terminal. This user has a terminal in its startup
items, and it is this terminal that runs the worker process. We do this because,
like TpWhateverWeCallItNow, we need a way to display an actual firefox for some
of the stone ridge tests.

### Processes running on the clients

#### srworker
{'fraction_non_alphanumeric': 0.04514829040992381, 'fraction_numerical': 0.006170896039292236, 'mean_word_length': 4.285191347753743, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5591590', 'n_tokens_mistral': 4242, 'n_tokens_neox': 3992, 'n_words': 2547}