<commit_before>Merge Scraper Data (for great justice!)
=======================================

There are a lot of consumer campaigns out there on the Internet. Consumer
campaigns supported by perfectly lovely organizations, organized around
causes you wholeheartedly support, that would change the world if enough
people followed through on them.

But it's hard to put them into practice. And it's *really* hard to put more
than one campaign at a time into practice. Different campaigns have different
scoring systems, different names for the same company, and, when they have
them at all, different apps that don't talk to each other.

``msd`` takes messy data from a bunch of different consumer campaigns, and
puts it into a single unified format.

``msd`` currently powers `SpendRight <http://spendright.org/search>`__
(``msd``'s creator) and the `thinkContext <http://thinkcontext.org>`__ browser
extension.


Using the data
==============

If you're not there already, check out
`msd's morph.io page <https://morph.io/spendright/msd>`__, where you can
view and download data merged from SpendRight's scrapers.

Keep in mind that the original consumer campaigns are generally copyrighted by
the non-profits that created them, and they have all sorts of different
terms/licensing agreements. It's up to you to decide whether to ask
them for permission now, or forgiveness later.

(This mostly applies to the ``claim`` and ``rating`` tables; facts about
companies and brands are almost certainly fair game.)


Installation
============

It's on PyPI: ``pip install msd``


Usage
=====

``msd db1.sqlite [db2.sqlite ...]``

This produces a file named ``msd.sqlite`` (you can change this with the ``-o``
switch).

If you don't have the library installed (e.g. for development), you
can use ``python -m msd.cmd`` in place of ``msd``.


Data format
===========
<commit_msg>update README to mention YAML<commit_after>Merge Scraper Data (for great justice!)
=======================================

There are a lot of consumer campaigns out there on the Internet. Consumer
campaigns supported by perfectly lovely organizations, organized around
causes you wholeheartedly support, that would change the world if enough
people followed through on them.

But it's hard to put them into practice. And it's *really* hard to put more
than one campaign at a time into practice. Different campaigns have different
scoring systems, different names for the same company, and, when they have
them at all, different apps that don't talk to each other.

``msd`` takes messy data from a bunch of different consumer campaigns, and
puts it into a single unified format.

``msd`` currently powers `SpendRight <http://spendright.org/search>`__
(``msd``'s creator) and the `thinkContext <http://thinkcontext.org>`__ browser
extension.


Using the data
==============

If you're not there already, check out
`msd's morph.io page <https://morph.io/spendright/msd>`__, where you can
view and download data merged from SpendRight's scrapers.

Keep in mind that the original consumer campaigns are generally copyrighted by
the non-profits that created them, and they have all sorts of different
terms/licensing agreements. It's up to you to decide whether to ask
them for permission now, or forgiveness later.

(This mostly applies to the ``claim`` and ``rating`` tables; facts about
companies and brands are almost certainly fair game.)


Installation
============

It's on PyPI: ``pip install msd``


Usage
=====

``msd db1.sqlite [db2.sqlite ...]``

This produces a file named ``msd.sqlite`` (you can change this with the ``-o``
switch).

``msd`` can also take YAML files as input. The YAML files should encode a
map from table name to list of rows (which are maps from column name to value).
For example::

  campaign:
  - author: <NAME>
    campaign_id: greenpeace_palm_oil
  rating:
  - campaign_id: greenpeace_palm_oil
    company: Colgate-Palmolive
    judgment: -1
  - campaign_id: greenpeace_palm_oil
    company: Danone
    judgment: 0

If you don't have the library installed (e.g. for development), you
can use ``python -m msd.cmd`` in place of ``msd``.


Data format
===========

{'fraction_non_alphanumeric': 0.11257801248199711, 'fraction_numerical': 0.0014402304368698992, 'mean_word_length': 4.623481781376518, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2786035', 'n_tokens_mistral': 1184, 'n_tokens_neox': 1105, 'n_words': 570}