<commit_before>##  How To Shavar-List

### Key Ideas

- Shavar implements [SafeBrowsing
  v2](https://web.archive.org/web/20160422212049/https://developers.google.com/safe-browsing/developers_guide_v2#Overview)
  to serve Mozilla-controlled lists of urls.
- Mozilla-controlled lists are sourced in the mozilla-services org on Github.
  E.g., [trackware lists provided by
  Disconnect](https://github.com/mozilla-services/shavar-prod-lists) and
  [plugin
  blocklists](https://github.com/mozilla-services/shavar-plugin-blocklist).
- Every 30m, Jenkins runs our
  [shavar-list-creation/lists2safebrowsing](https://github.com/mozilla-services/shavar-list-creation)
  script to convert the list files on GitHub into SafeBrowsing lists and upload
  them to s3.
- shavar serves lists from s3 as configured in the
  [`shavar-server-list-config`](https://github.com/mozilla-services/shavar-server-list-config)
  repo.

### Adding a new list

Shavar is a small, specific, efficient service for serving lists of urls to
Firefox. We do not plan to enhance its functionality much, except to make the
service easier to maintain and run. So if your project needs something more
than lists of urls, you might look at another service like
[Kinto](https://github.com/Kinto/kinto).

If you just need a list of urls ...

1. [File an issue](https://github.com/mozilla-services/shavar/issues/new) in
this repo, including a link back to any relevant bugzilla bug
2. We will ask you to add "upstream" files to an appropriate repository (e.g.,
   shavar-plugin-blocklist, shavar-prod-lists, or a new repo.)
3. We will add the lists to the staging section of our
   [shavar-list-creation](https://github.com/mozilla-services/shavar-list-creation)
   and/or
   [shavar-list-creation-config](https://github.com/mozilla-services/shavar-list-creation-config)
   repositories.
4. When we verify that we're creating and publishing the SafeBrowsing lists to
   S3, we will add the names of the lists to the staging section of
   [shavar-server-list-config](https://github.com/mozilla-services/shavar-server-list-config)
   to make shavar start serving the lists.
5. When we verify that shavar is serving the lists to Firefox correctly, we
   will repeat steps 3-4 for production.

### Updating list contents<commit_msg>add QA section<commit_after>##  How To Shavar-List

### Key Ideas

- Shavar implements [SafeBrowsing
  v2](https://web.archive.org/web/20160422212049/https://developers.google.com/safe-browsing/developers_guide_v2#Overview)
  to serve Mozilla-controlled lists of urls.
- Mozilla-controlled lists are sourced in the mozilla-services org on Github.
  E.g., [trackware lists provided by
  Disconnect](https://github.com/mozilla-services/shavar-prod-lists) and
  [plugin
  blocklists](https://github.com/mozilla-services/shavar-plugin-blocklist).
- Every 30m, Jenkins runs our
  [shavar-list-creation/lists2safebrowsing](https://github.com/mozilla-services/shavar-list-creation)
  script to convert the list files on GitHub into SafeBrowsing lists and upload
  them to s3.
- shavar serves lists from s3 as configured in the
  [`shavar-server-list-config`](https://github.com/mozilla-services/shavar-server-list-config)
  repo.

### Adding a new list

Shavar is a small, specific, efficient service for serving lists of urls to
Firefox. We do not plan to enhance its functionality much, except to make the
service easier to maintain and run. So if your project needs something more
than lists of urls, you might look at another service like
[Kinto](https://github.com/Kinto/kinto).

So, if you just need a list of urls ...

1. [File an issue](https://github.com/mozilla-services/shavar/issues/new) in
this repo, including a link back to any relevant bugzilla bug
2. We will ask you to add "upstream" files to an appropriate repository (e.g.,
   shavar-plugin-blocklist, shavar-prod-lists, or a new repo.)
3. We will add the lists to the staging section of our
   [shavar-list-creation](https://github.com/mozilla-services/shavar-list-creation)
   and/or
   [shavar-list-creation-config](https://github.com/mozilla-services/shavar-list-creation-config)
   repositories.
4. When we verify that we're creating and publishing the SafeBrowsing lists to
   S3, we will add the names of the lists to the staging section of
   [shavar-server-list-config](https://github.com/mozilla-services/shavar-server-list-config)
   to make shavar start serving the lists.
5. When we verify that shavar is serving the lists to Firefox correctly, we
   will repeat steps 3-4 for production.

#### QA

When we add new lists to shavar,
[Firefox Test Engineering](https://readthedocs.org/projects/firefox-test-engineering/)
verifies that lists do not disrupt the server-side shavar service. We will
always add the new lists to the staging server first, as described above.

If you need help with client-side testing of Firefox behavior with the new
list, contact the
[SoftVision QA Team](https://wiki.mozilla.org/QA_SoftVision_Team). Typically,
you will need to provide a set of new or updated testing prefs in the
[`services-test/shavar/e2e-test/prefs.ini`](https://github.com/mozilla-services/services-test/blob/master/shavar/e2e-test/prefs.ini)
file.

### Updating list contents
{'fraction_non_alphanumeric': 0.09753769803397595, 'fraction_numerical': 0.011452567283832793, 'mean_word_length': 4.198412698412699, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 25, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2312157', 'n_tokens_mistral': 1694, 'n_tokens_neox': 1593, 'n_words': 559}