<commit_before># FingerprintSecureDrop

The code herein can be used to collect Tor cell traces, which can be used to
train classifiers to website fingerprint SecureDrop (SD) instances. The idea is
to use this as a way to test server-side defense mitigations we hope to
implement. If you're interested in website fingerprinting and defenses
(especially in relation to Tor), please feel encouraged to come talk with a
group of researchers in a Gitter chatroom at
https://gitter.im/freedomofpress/Website_Fingerprinting. 

We're hoping to the code in this repo a little more composable soon, so you can
make some modifications to get this going for your own onion service (or perhaps
even regular site). Until then, you could still try it out on SecureDrop, and
then analyze the traces with your favorite classifiers. If you do so, please
share your results with us! (Preferably, using PGP.)

`sort_onions.py` first scrapes onion service URLs from some popular onion
service indexes, and then scrapes each of those .onion URLs. If it encounters an
up-to-date SD, it stores it as such. If it encounters a site that doesn't
mention SD at all, it also stores that URL in a different set. These are the two
sets that really matter (i.e., are used by `crawl_onions.py`). Out-of-date SDs
could mess with our results (shame on you admins ;-P), and we also ignore sites
that even mention SD as a precaution against sites like Greenpeace's heavily
modified, very out-of-date SecureDrop that they seem to have rebranded as
"SafeSource," but fortunately still includes the string "SecureDrop" on the
homepage. Thus we can catch it with this precautionary measure. Hopefully, no
sites have made minor changes to SecureDrop and also dropped mention of
SecureDrop, because that would have potential to mess up our results. I believe
that's probably unlikely.

`crawl_onions.py` crawls onion services roughly following Tao and Goldberg (see
https://cypherpunks.ca/~iang/pubs/webfingerprint-wpes.pdf, Appendix A, Algorithm
1). Collecting a trace for a single site happens roughly as follows:

* Load single onion service in Tor Browser (using a virtual framebuffer for
    headless operation by default)
* Wait 5 (configurable) seconds after the initial page onload event to capture
    traffic after the initial onload event
* Use stem to identify the unique, singular Tor circuit id for the rendezvous
    circuit to meet the onion service we're loading
* Read the Tor cell log from our modified `tor` binary and collects all DATA
    cells from that circuit id
    * Write these as `(time, direction)` pairs in a file on disk, adjusted so
        the first cell is time zero
* Close all circuits with stem
* Pause 5s (we're currently experimenting with this value to see if it's
  necessary and how it affects our results)

The adversary model we are assuming is as follows. First, our adversary is in
control of our guard node. Thus, they are able to separate individual streams
(rendezvous circuits are actually not multiplexed within TCP streams like
general circuits, but I mention this because it's relevant to general circuits,
which are multiplexed). Second, our adversary has read Kwon et. al's circuit
fingerprint paper (see
https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-kwon.pdf)
and is capable of fingerprinting our rendezvous circuits with 100% accuracy. The
way we collect traces described above is representative of the capabilities of
such an adversary.

We have begun to use a fixed entry node because it seems that a malicious actor
in control of one or more guard nodes would train a separate classifier on each
guard node for fingerprinting the traffic that goes through that guard node.
Training on data produced under the same network conditions as you would wish to
perform real-world attacks should produce more accurate results.

Lastly, `config.ini` is a general configuration file, for those who just want to
run the script and tinker with some parameters. This should be expanded in the
future to expose even more functionality and make it easy to swap out your own
website for SD in order to test fingerprinting defenses on your own hidden
service.

## Getting Started

This repo uses submodules, so use `git clone --recursive`.

* Requires Ansible >= 2.0
* Requires Vagrant

To get a properly configured VM bootstrapped for crawling, run:

```
cd FingerprintSecureDrop
vagrant up
```

We seriously recommend using the Vagrantfile and Ansible playbook, which will
sping up a VM and provision it for you, it is not strictly necessary. A lot of
things are necessary to get this to work otherwise including compiling a custom
version of Tor. We will not be providing instructions for how to do this, but
the patched `relay.c` is here and everything you'd need to do to get this code
running can be figured out by reading through the Ansible playbook.

### Running the Sorter

```
vagrant ssh
cd ~/FingerprintSecureDrop/fpsd
./sorter.py
```

To look at the sorter log while it's running run `less +F
logging/sorter-latest.log`. Sorter data will be timestamped with
`logging/class-data-latest.pickle` being symlinked to the latest data
(timestamping and symlinking like this is done with all logs and data files that
are created by the Python processes in this repo).

### Running the Crawler

```
vagrant ssh
cd ~/FingerprintSecureDrop/fpsd
./crawler.py
```

To look at the crawler log while it's running run `less +F
logging/crawler-latest.log`, and to look at the raw Tor cell log run `less +F
logging/tor_cell_seq.log`. You can also check out the traces it's collecting as
it runs: `cd logging/batch-latest`.

A systemd unit is also provided to run the crawler on repeat. Simply run
`systemctl start crawler@<your username>.service` to start the crawler running
on repeat.

### Deploying to remote servers

Todo: write
<commit_msg>Suggest specific versions of Ansible, VirtualBox, and Vagrant I've tested with<commit_after># FingerprintSecureDrop

The code herein can be used to collect Tor cell traces, which can be used to
train classifiers to website fingerprint SecureDrop (SD) instances. The idea is
to use this as a way to test server-side defense mitigations we hope to
implement. If you're interested in website fingerprinting and defenses
(especially in relation to Tor), please feel encouraged to come talk with a
group of researchers in a Gitter chatroom at
https://gitter.im/freedomofpress/Website_Fingerprinting. 

We're hoping to the code in this repo a little more composable soon, so you can
make some modifications to get this going for your own onion service (or perhaps
even regular site). Until then, you could still try it out on SecureDrop, and
then analyze the traces with your favorite classifiers. If you do so, please
share your results with us! (Preferably, using PGP.)

`sort_onions.py` first scrapes onion service URLs from some popular onion
service indexes, and then scrapes each of those .onion URLs. If it encounters an
up-to-date SD, it stores it as such. If it encounters a site that doesn't
mention SD at all, it also stores that URL in a different set. These are the two
sets that really matter (i.e., are used by `crawl_onions.py`). Out-of-date SDs
could mess with our results (shame on you admins ;-P), and we also ignore sites
that even mention SD as a precaution against sites like Greenpeace's heavily
modified, very out-of-date SecureDrop that they seem to have rebranded as
"SafeSource," but fortunately still includes the string "SecureDrop" on the
homepage. Thus we can catch it with this precautionary measure. Hopefully, no
sites have made minor changes to SecureDrop and also dropped mention of
SecureDrop, because that would have potential to mess up our results. I believe
that's probably unlikely.

`crawl_onions.py` crawls onion services roughly following Tao and Goldberg (see
https://cypherpunks.ca/~iang/pubs/webfingerprint-wpes.pdf, Appendix A, Algorithm
1). Collecting a trace for a single site happens roughly as follows:

* Load single onion service in Tor Browser (using a virtual framebuffer for
    headless operation by default)
* Wait 5 (configurable) seconds after the initial page onload event to capture
    traffic after the initial onload event
* Use stem to identify the unique, singular Tor circuit id for the rendezvous
    circuit to meet the onion service we're loading
* Read the Tor cell log from our modified `tor` binary and collects all DATA
    cells from that circuit id
    * Write these as `(time, direction)` pairs in a file on disk, adjusted so
        the first cell is time zero
* Close all circuits with stem
* Pause 5s (we're currently experimenting with this value to see if it's
  necessary and how it affects our results)

The adversary model we are assuming is as follows. First, our adversary is in
control of our guard node. Thus, they are able to separate individual streams
(rendezvous circuits are actually not multiplexed within TCP streams like
general circuits, but I mention this because it's relevant to general circuits,
which are multiplexed). Second, our adversary has read Kwon et. al's circuit
fingerprint paper (see
https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-kwon.pdf)
and is capable of fingerprinting our rendezvous circuits with 100% accuracy. The
way we collect traces described above is representative of the capabilities of
such an adversary.

We have begun to use a fixed entry node because it seems that a malicious actor
in control of one or more guard nodes would train a separate classifier on each
guard node for fingerprinting the traffic that goes through that guard node.
Training on data produced under the same network conditions as you would wish to
perform real-world attacks should produce more accurate results.

Lastly, `config.ini` is a general configuration file, for those who just want to
run the script and tinker with some parameters. This should be expanded in the
future to expose even more functionality and make it easy to swap out your own
website for SD in order to test fingerprinting defenses on your own hidden
service.

## Getting Started

This repo uses submodules, so use `git clone --recursive`. You will also need to
install the following:

* Ansible 1.9.6
* Vagrant 1.8.5
* VirtualBox 5.1.4

To get a properly configured VM bootstrapped for crawling, run:

```
cd FingerprintSecureDrop
vagrant up
```

We seriously recommend using the Vagrantfile and Ansible playbook, which will
sping up a VM and provision it for you, it is not strictly necessary. A lot of
things are necessary to get this to work otherwise including compiling a custom
version of Tor. We will not be providing instructions for how to do this, but
the patched `relay.c` is here and everything you'd need to do to get this code
running can be figured out by reading through the Ansible playbook.

### Running the Sorter

```
vagrant ssh
cd ~/FingerprintSecureDrop/fpsd
./sorter.py
```

To look at the sorter log while it's running run `less +F
logging/sorter-latest.log`. Sorter data will be timestamped with
`logging/class-data-latest.pickle` being symlinked to the latest data
(timestamping and symlinking like this is done with all logs and data files that
are created by the Python processes in this repo).

### Running the Crawler

```
vagrant ssh
cd ~/FingerprintSecureDrop/fpsd
./crawler.py
```

To look at the crawler log while it's running run `less +F
logging/crawler-latest.log`, and to look at the raw Tor cell log run `less +F
logging/tor_cell_seq.log`. You can also check out the traces it's collecting as
it runs: `cd logging/batch-latest`.

A systemd unit is also provided to run the crawler on repeat. Simply run
`systemctl start crawler@<your username>.service` to start the crawler running
on repeat.

### Deploying to remote servers

Todo: write

{'fraction_non_alphanumeric': 0.047217537942664416, 'fraction_numerical': 0.0026138279932546374, 'mean_word_length': 4.448323380799265, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6451597', 'n_tokens_mistral': 3234, 'n_tokens_neox': 3022, 'n_words': 1813}