<commit_before>Introduction
------------

Provides a Web server for symbolicating Firefox stacks. It matches PC addresses
to modules in memory and looks up the corresponding function names in
server-side symbol files (.SYM files).

This project is intended as a drop-in replacement for the old
[Snappy Symbolication Server](https://github.com/mozilla/Snappy-Symbolication-Server/tree/e38f3fdd7b49ab181a95bd76bbd3a5f4fea97276)

Quick Start without Docker
--------------------------

1. Install [Python 2.7](https://www.python.org/downloads/) and
   [pip](https://pip.pypa.io/en/stable/installing/) if they are not already
   installed. Pip must be version 8.0 or above. It can be upgraded using
   `pip install -U 'pip>=8'`.
2. Install dependencies: `pip install -r requirements.txt`. You may want to do
   this in a
   [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
3. Install [memcached](http://www.memcached.org/downloads)
4. Copy/create a configuration file and set values appropriately. See
   [Configuration File](#configuration-file)
5. Run `python quickstart.py -c [configuration file]`

The server(s) can then be stopped with
`python quickstart.py -c [configuration file] --stop`.
This will only work if quickstart was used to start the server(s).

Quick Start with Docker
-----------------------

1. Install [Docker](https://docs.docker.com/engine/installation)
2. It may be necessary that your user be in the Docker group. This can be
   accomplished in Linux with `sudo usermod -aG docker $(whoami)`. You must log
   out for changes to take effect.
3. Install [Python 2.7](https://www.python.org/downloads/) and
   [pip](https://pip.pypa.io/en/stable/installing/) if they are not already
   installed. Pip must be version 8.0 or above. It can be upgraded using
   `pip install -U 'pip>=8'`.
4. Install dependencies: `pip install -r requirements.txt`. You may want to do
   this in a
   [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
5. Copy/create a configuration file and set values appropriately. See
   [Configuration File](#configuration-file). To use Docker, it is important
   that the `quickstart.Docker.enable = true`.
6. Run `python quickstart.py -c [configuration file]`

Quickstart attempts to do some caching in order to recreate the image and
container only when necessary. The image will be built only once. The container
will be recreated any time that the passed configuration changes. You can
force a full rebuild with `python quickstart.py -c [configuration file] -R`.
A full rebuild should be done anytime that the code changes.

The docker container can be stopped with
`python quickstart.py -c [configuration file] --stop`.

DiskCache
---------

An LRU cache of symbolication files. On cache misses, the file is automatically
retrieved and added to the cache.

**Important:** The directory used for cache data (specified in the configuration
as `DiskCache.cachePath`) must be used only for cache data. When DiskCache
starts, it reads all files in the cache directory into the cache. If there is
anything else in the cache path, it eventually will be evicted from the cache
and deleted from the disk.

**Also Important:** Do not delete anything from the cache while the DiskCache is
running. Doing this will screw up the cache. If DiskCache files get deleted,
restart the DiskCache server.

SymServer communicates with the DiskCache with the same protocol used to
make requests of SymServer.

The DiskCache imposes a size limit specified by the configuration option
`DiskCache.maxSizeMB`. However, the size limit is not strictly observed. When a
file is added to the cache, it is saved to the cache directory BEFORE evicting
enough cache items to bring the cache back under its maximum size.

Symbol files can be fairly large. Currently xul.sym is 72MB. However, before
being saved to the disk, the symbol data is processed to remove unnecessary data
and format remaining data into a more easily searchable form. After processing
xul.sym, only 13MB of data remain. This allows the cache to contain many symbols
more than it would otherwise.

SymServer
---------

Utilizes two caches to quickly symbolicate stacks. The first cache is memcached.
Addresses not in memcached are requested from DiskCache, which will either
retrieve the symbols from the cache, or download the symbols, add them to the
cache, and return them.

Takes POST requests in JSON format. This is the standard symbolication request
for testing:

```
curl -d '{"stacks":[[[0,11723767],[1, 65802]]],"memoryMap":[["xul.pdb","44E4EC8C2F41492B9369D6B9A059577C2"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]],"version":4}' 127.0.0.1:8080
```

And the corresponding response:

```
{"symbolicatedStacks": [["XREMain::XRE_mainRun() (in xul.pdb)", "KiUserCallbackDispatcher (in wntdll.pdb)"]], "knownModules": [true, true]}
```

Note that while it is possible to run the SymServer without memcached, DiskCache
is required for SymServer to operate properly.

Quickstart
----------

The quickstart script is designed to make it easier to start several servers at
once. The script keeps track of running instances by keeping pidfiles in a
directory called "pids" in the same directory as the quickstart script.
**Note:** If a server is started in any way other than by quickstart, its
pidfile will not be written to the correct place and quickstart will assume that
the server is not running and may try to start it again.

Quickstart's behavior is controlled largely by the configuration file. The
configuration file must be passed to quickstart for it to start the servers. The
configuration file passed will then be used by the servers that it starts as
well. For more information about how to pass in a configuration file, use the
`--help` argument.

Note that although there is a memcached binary available for Windows, quickstart
does not support it because of differences in operation and command line<commit_msg>Document the request and response format (#66)<commit_after>Introduction
------------

Provides a Web server for symbolicating Firefox stacks. It matches function offsets
to modules in memory and looks up the corresponding function names in
server-side symbol files in [Breakpad's .sym format](https://chromium.googlesource.com/breakpad/breakpad/+/master/docs/symbol_files.md).

This project is intended as a drop-in replacement for the old
[Snappy Symbolication Server](https://github.com/mozilla/Snappy-Symbolication-Server/tree/e38f3fdd7b49ab181a95bd76bbd3a5f4fea97276).

The canonical Mozilla instance of this server runs at `http://symbolapi.mozilla.org/`.

Quick Start without Docker
--------------------------

1. Install [Python 2.7](https://www.python.org/downloads/) and
   [pip](https://pip.pypa.io/en/stable/installing/) if they are not already
   installed. Pip must be version 8.0 or above. It can be upgraded using
   `pip install -U 'pip>=8'`.
2. Install dependencies: `pip install -r requirements.txt`. You may want to do
   this in a
   [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
3. Install [memcached](http://www.memcached.org/downloads)
4. Copy/create a configuration file and set values appropriately. See
   [Configuration File](#configuration-file)
5. Run `python quickstart.py -c [configuration file]`

The server(s) can then be stopped with
`python quickstart.py -c [configuration file] --stop`.
This will only work if quickstart was used to start the server(s).

Quick Start with Docker
-----------------------

1. Install [Docker](https://docs.docker.com/engine/installation)
2. It may be necessary that your user be in the Docker group. This can be
   accomplished in Linux with `sudo usermod -aG docker $(whoami)`. You must log
   out for changes to take effect.
3. Install [Python 2.7](https://www.python.org/downloads/) and
   [pip](https://pip.pypa.io/en/stable/installing/) if they are not already
   installed. Pip must be version 8.0 or above. It can be upgraded using
   `pip install -U 'pip>=8'`.
4. Install dependencies: `pip install -r requirements.txt`. You may want to do
   this in a
   [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
5. Copy/create a configuration file and set values appropriately. See
   [Configuration File](#configuration-file). To use Docker, it is important
   that the `quickstart.Docker.enable = true`.
6. Run `python quickstart.py -c [configuration file]`

Quickstart attempts to do some caching in order to recreate the image and
container only when necessary. The image will be built only once. The container
will be recreated any time that the passed configuration changes. You can
force a full rebuild with `python quickstart.py -c [configuration file] -R`.
A full rebuild should be done anytime that the code changes.

The docker container can be stopped with
`python quickstart.py -c [configuration file] --stop`.

DiskCache
---------

An LRU cache of symbolication files. On cache misses, the file is automatically
retrieved and added to the cache.

**Important:** The directory used for cache data (specified in the configuration
as `DiskCache.cachePath`) must be used only for cache data. When DiskCache
starts, it reads all files in the cache directory into the cache. If there is
anything else in the cache path, it eventually will be evicted from the cache
and deleted from the disk.

**Also Important:** Do not delete anything from the cache while the DiskCache is
running. Doing this will screw up the cache. If DiskCache files get deleted,
restart the DiskCache server.

SymServer communicates with the DiskCache with the same protocol used to
make requests of SymServer.

The DiskCache imposes a size limit specified by the configuration option
`DiskCache.maxSizeMB`. However, the size limit is not strictly observed. When a
file is added to the cache, it is saved to the cache directory BEFORE evicting
enough cache items to bring the cache back under its maximum size.

Symbol files can be fairly large. Currently xul.sym is 72MB. However, before
being saved to the disk, the symbol data is processed to remove unnecessary data
and format remaining data into a more easily searchable form. After processing
xul.sym, only 13MB of data remain. This allows the cache to contain many symbols
more than it would otherwise.

SymServer
---------

Utilizes two caches to quickly symbolicate stacks. The first cache is memcached.
Addresses not in memcached are requested from DiskCache, which will either
retrieve the symbols from the cache, or download the symbols, add them to the
cache, and return them.

Takes POST requests in JSON format. This is the standard symbolication request
for testing:

```
curl -d '{"stacks":[[[0,11723767],[1, 65802]]],"memoryMap":[["xul.pdb","44E4EC8C2F41492B9369D6B9A059577C2"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]],"version":4}' 127.0.0.1:8080
```

The request body must be a JSON object with the following three properties:
* `version`: the version number of the request format as an integer. Currently `4`.
* `stacks`: an array of stack traces. Each stack trace is an array of frames. Each frame is an array of two elements: `[module_index, module_offset]`. `module_index` is an index into the array in the `memoryMap` property as an integer. `module_offset` is the offset of this frame's instruction pointer relative to the base memory address of the module in which the code is contained as an integer.
* `memoryMap`: an array of modules. Each module is an array of two strings: `[debug_file, debug_identifier]`. Fully describing these parameters is out of scope for this document, but they are the last two elements of the `MODULE` line of a symbol file, and they can be used to locate the matching symbol file.

And the corresponding response:

```
{"symbolicatedStacks": [["XREMain::XRE_mainRun() (in xul.pdb)", "KiUserCallbackDispatcher (in wntdll.pdb)"]], "knownModules": [true, true]}
```

The response body is a JSON object with the following two properties:
* `symbolicatedStacks`: an array of stack traces, matching the order of the entries of the `stacks` property in the request. Each stack trace is an array of strings containing the function name for this stack frame, if available.
* `knownModules`: an array of booleans matching the order of the entries of the `memoryMap` property in the request. Each entry will be `true` if symbols were found for this module, and `false` otherwise.


Note that while it is possible to run the SymServer without memcached, DiskCache
is required for SymServer to operate properly.

Quickstart
----------

The quickstart script is designed to make it easier to start several servers at
once. The script keeps track of running instances by keeping pidfiles in a
directory called "pids" in the same directory as the quickstart script.
**Note:** If a server is started in any way other than by quickstart, its
pidfile will not be written to the correct place and quickstart will assume that
the server is not running and may try to start it again.

Quickstart's behavior is controlled largely by the configuration file. The
configuration file must be passed to quickstart for it to start the servers. The
configuration file passed will then be used by the servers that it starts as
well. For more information about how to pass in a configuration file, use the
`--help` argument.

Note that although there is a memcached binary available for Windows, quickstart
does not support it because of differences in operation and command line
{'fraction_non_alphanumeric': 0.08298908232516966, 'fraction_numerical': 0.01630274417232222, 'mean_word_length': 4.204222648752399, 'pattern_counts': {'":': 10, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 13, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3645681', 'n_tokens_mistral': 3784, 'n_tokens_neox': 3560, 'n_words': 1856}