<commit_before>
.. _people:

Contributing a Municipal Person Scraper
=======================================

This document is meant to provide a tutorial-like overview of the steps toward
contributing a munipal Person scraper to the Open Civic Data project.

This guide assumes you have a working pupa setup. If you don't please
look into the intro / quickstart guide.

.. TODO:: HELP WITH LINKING TO INTRO


Creating a New Person scraper
-----------------------------

Great. So, let's get started. A scraper class (such as a PersonScraper) may
live anywhere, but it's common to break each scraper into it's own file, to
avoid having one massive file.

The pattern is usually to create a `person.py` file next to your Jurisdiction's
`__init__.py`, and create a `FooPersonScraper`, where the string `Foo` is a
nicely human readable slug of your Jurisdiction (such as `BostonPersonScraper`
or `NewYorkCityPersonScraper`.)

This scraper class shall inherit from the `Scraper` class, found in
the `pupa.scrape` module.

The following is a short example::

    from pupa.scrape import Scraper
    class MyFirstPersonScraper(Scraper):
        pass

This class won't do anything, and should even result in an error. Let's iterate
on this concept to work up to something useful.

Every `Person` scraper inherets a `scrape_people` method. Usually it's not
advised to override this method, rather, implementing a proper
`get_people` method (which will `yield` back `Person` objects to `scrape_people`
to save to disk) is the correct way to write a scraper.

You may also yield an iterable of `Person` objects, which helps if you
are scraping both people and committees for the Jurisdiction, but want
to keep the scraper logic in their own routines.

As you might have guessed by now, `Person` scrapers scrape many `People`, as
well as any `Membership` objects that you might find along the way.

Let's take a look at a dead-simple Pupa scraper::

    from pupa.scrape import Scraper, Legislator
    class MyFirstPersonScraper(Scraper):
        def get_people(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")
            yield js

You can see that we create the Legislator, with the only two required
params (`name` and `post_id`, add the source of the data (most of the time
this will be the url that you've called with `urlopen`) and yielded the
Legislator back.

Right. Now let's get back to Memberships. Let's say that we've found
that <NAME> has a membership in the Transportation committee::

    from pupa.scrape import Scraper, Legislator
    class MyFirstPersonScraper(Scraper):
        def get_people(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")
            js.add_committee_membership("Transportation",
                                        role="Chair")
            yield js

Of course, all of this is well and good if we find all the data on the
same page. However, commonly, it's much easier to scrape each committee
from the committee pages, since this will often have the data in
an easier-to-scrape format.

Rather than write something like::

    from pupa.scrape import Scraper, Legislator, Committee
    class MyFirstPersonScraper(Scraper):
        def get_people(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")

            members = ["<NAME>", "<NAME>"]

            committee = Committee("Transportation")
            committee.add_source("http://example.com/committee/transport")
            for member in members:
                committee.add_member(member, role='member')

            yield committee
            yield js

However, as you can imagine, this gets quite out of hand quite quickly. One
common pattern is to split the logic into two sections, such as::


    from pupa.scrape import Scraper, Legislator, Committee
    class MyFirstPersonScraper(Scraper):
        def scrape_legislators(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")
            yield js

        def scrape_committees(self):
            members = ["<NAME>", "<NAME>"]
            committee = Committee("Transportation")
            committee.add_source("http://example.com/committee/transport")
            for member in members:
                committee.add_member(member, role='member')

            yield committee

        def get_people(self):
            yield self.scrape_legislators()
            yield self.scrape_committees()

It's worth noting that you should keep in mind `scrape_people` *is* a special
function (see above), so you should take care not to override this method.<commit_msg>Add better person scraping Example<commit_after>
.. _people:

Contributing a Municipal Person Scraper
=======================================

This document is meant to provide a tutorial-like overview of the steps toward
contributing a munipal Person scraper to the Open Civic Data project.

This guide assumes you have a working pupa setup. If you don't please
look into the intro / quickstart guide.

.. TODO:: HELP WITH LINKING TO INTRO


Creating a New Person scraper
-----------------------------

Great. So, let's get started. A scraper class (such as a PersonScraper) may
live anywhere, but it's common to break each scraper into it's own file, to
avoid having one massive file.

The pattern is usually to create a `person.py` file next to your Jurisdiction's
`__init__.py`, and create a `FooPersonScraper`, where the string `Foo` is a
nicely human readable slug of your Jurisdiction (such as `BostonPersonScraper`
or `NewYorkCityPersonScraper`.)

This scraper class shall inherit from the `Scraper` class, found in
the `pupa.scrape` module.

The following is a short example::

    from pupa.scrape import Scraper
    class MyFirstPersonScraper(Scraper):
        pass

This class won't do anything, and should even result in an error. Let's iterate
on this concept to work up to something useful.

Every `Person` scraper inherets a `scrape_people` method. Usually it's not
advised to override this method, rather, implementing a proper
`get_people` method (which will `yield` back `Person` objects to `scrape_people`
to save to disk) is the correct way to write a scraper.

You may also yield an iterable of `Person` objects, which helps if you
are scraping both people and committees for the Jurisdiction, but want
to keep the scraper logic in their own routines.

As you might have guessed by now, `Person` scrapers scrape many `People`, as
well as any `Membership` objects that you might find along the way.

Let's take a look at a dead-simple Pupa scraper::

    from pupa.scrape import Scraper, Legislator
    class MyFirstPersonScraper(Scraper):
        def get_people(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")
            yield js

You can see that we create the Legislator, with the only two required
params (`name` and `post_id`, add the source of the data (most of the time
this will be the url that you've called with `urlopen`) and yielded the
Legislator back.

Right. Now let's get back to Memberships. Let's say that we've found
that <NAME> has a membership in the Transportation committee::

    from pupa.scrape import Scraper, Legislator
    class MyFirstPersonScraper(Scraper):
        def get_people(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")
            js.add_committee_membership("Transportation",
                                        role="Chair")
            yield js

Of course, all of this is well and good if we find all the data on the
same page. However, commonly, it's much easier to scrape each committee
from the committee pages, since this will often have the data in
an easier-to-scrape format.

Rather than write something like::

    from pupa.scrape import Scraper, Legislator, Committee
    class MyFirstPersonScraper(Scraper):
        def get_people(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")

            members = ["<NAME>", "<NAME>"]

            committee = Committee("Transportation")
            committee.add_source("http://example.com/committee/transport")
            for member in members:
                committee.add_member(member, role='member')

            yield committee
            yield js

However, as you can imagine, this gets quite out of hand quite quickly. One
common pattern is to split the logic into two sections, such as::


    from pupa.scrape import Scraper, Legislator, Committee
    class MyFirstPersonScraper(Scraper):
        def scrape_legislators(self):
            js = Legislator(name="<NAME>", post_id="Ward 1")
            js.add_source(url="http://example.com")
            yield js

        def scrape_committees(self):
            members = ["<NAME>", "<NAME>"]
            committee = Committee("Transportation")
            committee.add_source("http://example.com/committee/transport")
            for member in members:
                committee.add_member(member, role='member')

            yield committee

        def get_people(self):
            yield self.scrape_legislators()
            yield self.scrape_committees()

It's worth noting that you should keep in mind `scrape_people` *is* a special
function (see above), so you should take care not to override this method.

Of course, in real scrapers, you'll need to write some code to take care
of getting the list of people that are in that jurisdiction, or have
memberships in the Legislature. Hardcoding names, such as in the examples
above is bad practice, since it's usually easier to just manually enter
in the data.

As a slightly more fun example, here's a scraper that will scrape the
Sunlight website for people's information. This is deliberately a mildly
complex example (as well as being purely for fun!), to get a feel for what
a working Person scraper may look like::

    import re
    import lxml
    from pupa.scrape import Scraper, Legislator, Committee
    class SunlightPersonScraper(Scraper):
        def get_people(self):
            url = "http://sunlightfoundation.com/people/"
            entry = self.urlopen(url)
            page = lxml.html.fromstring(entry)
            page.make_links_absolute(url)
            for person in page.xpath("//ul[contains(@class, 'sunlightStaff')]//li"):
                who = person.xpath(".//a")
                who = who[0] if who else None
                name = who.text_content().strip()
                image = person.xpath(".//div[@class='imgWrapper']/@style")
                image = image[0] if image else None
                if image:
                    urls = re.findall("url\((?P<url>.*)\);", image)
                position = person.xpath(".//span[@class='staffTitle']/text()")[0]
                position = position.strip()
                person = Legislator(name=name,
                                    post_id=position,
                                    image=image)
                person.add_link('homepage', who.attrib['href'])
                person.add_source(url)
                yield person
{'fraction_non_alphanumeric': 0.09131697017682766, 'fraction_numerical': 0.000967713556787191, 'mean_word_length': 2.887824897400821, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '851738', 'n_tokens_mistral': 3185, 'n_tokens_neox': 2991, 'n_words': 1271}