<commit_before>Upton
==========
Upton is a framework for easy web-scraping with a useful debug mode that doesn't hammer your target's servers. It does the repetitive parts of writing scrapers, so you only have to write the unique parts for each site.

Documentation
----------------------

Upton works best if your scraper class inherits from Upton::Scraper, e.g. 

     class MyScraper < Upton::Scraper 
     ...
     end

Upton::Scraper is implemented as an abstract class. For basic use cases, you only have to implement one method -- the `get_index` method -- and a block argument for `scrape` to do something with the scraped pages. 

Upton operates on the theory that, for most scraping projects, you need to
scrape two types of pages:

1. Index pages, which list instance pages. For example, a job search 
    site's search page or a newspaper's homepage.
2. Instance pages, which represent the goal of your scraping, e.g.
    job listings or news articles.


In more complicated cases, you may need to write additional methods; for example, if you need to log in to a site before scraping the instance pages, you would need to override the scrape method in your subclass. 

You get, for free, methods like `get_page(url, stash=false)` which, well, gets a page. That's not very special. The more interesting part is that `get_page(url, stash=false)` transparently stashes the response of each request. Whenever you repeat a request with `true` as the second parameter, the stashed HTML is returned without going to the server. This is helpful in the development stages of a project when you're testing some aspect of the code and don't want to hit a server each time.

Upton also sleeps (by default) 30 seconds between non-stashed requests, to reduce load on the server you're scraping. This is configurable with the `@nice_sleep_time` option.

**Important Note:** Upton is alpha software. The API may change at any time and without warning. 

Example
----------------------
If you want to scrape ProPublica's website with Upton, this is how you'd do it. (Scraping our [RSS feed](http://feeds.propublica.org/propublica/main) would be smarter, but not every site has a full-text RSS feed...)


      class ProPublicaScraper < Upton::Scraper
        def get_index
          url = "http://www.propublica.org"
          doc = Nokogiri::HTML(self._get_page(url, true))
          doc.css("section#river section h1 a").to_a.map{|l| + l["href"] } #map the Nokogiri link elements to a string representation of a URL.
        end
      end

      ProPublicaScraper.new.scrape do |article_string|
        puts "here is the full text of the ProPublica article: \n #{article_string}"
        #or, do other stuff here.
      end


Why "Upton"
----------------------
<NAME> was a pioneering, muckraking journalist who is most famous for _The Jungle_, a novel portraying the reality of immigrant labor struggles in Chicago meatpacking plants at the start of the 1900s. Upton, the gem, sprang out of a ProPublica project pertaining to labor issues.

License (MIT)<commit_msg>add link to github pages documentation to README<commit_after>Upton
==========
Upton is a framework for easy web-scraping with a useful debug mode that doesn't hammer your target's servers. It does the repetitive parts of writing scrapers, so you only have to write the unique parts for each site.

Documentation
----------------------

Upton works best if your scraper class inherits from Upton::Scraper, e.g. 

     class MyScraper < Upton::Scraper 
     ...
     end

Upton::Scraper is implemented as an abstract class. For basic use cases, you only have to implement one method -- the `get_index` method -- and a block argument for `scrape` to do something with the scraped pages. 

Upton operates on the theory that, for most scraping projects, you need to
scrape two types of pages:

1. Index pages, which list instance pages. For example, a job search 
    site's search page or a newspaper's homepage.
2. Instance pages, which represent the goal of your scraping, e.g.
    job listings or news articles.


In more complicated cases, you may need to write additional methods; for example, if you need to log in to a site before scraping the instance pages, you would need to override the scrape method in your subclass. 

You get, for free, methods like `get_page(url, stash=false)` which, well, gets a page. That's not very special. The more interesting part is that `get_page(url, stash=false)` transparently stashes the response of each request. Whenever you repeat a request with `true` as the second parameter, the stashed HTML is returned without going to the server. This is helpful in the development stages of a project when you're testing some aspect of the code and don't want to hit a server each time.

Upton also sleeps (by default) 30 seconds between non-stashed requests, to reduce load on the server you're scraping. This is configurable with the `@nice_sleep_time` option.

**For more complete documentation**, see [our Rocco docs](http://propublica.github.io/upton).

**Important Note:** Upton is alpha software. The API may change at any time and without warning. 

Example
----------------------
If you want to scrape ProPublica's website with Upton, this is how you'd do it. (Scraping our [RSS feed](http://feeds.propublica.org/propublica/main) would be smarter, but not every site has a full-text RSS feed...)


      class ProPublicaScraper < Upton::Scraper
        def get_index
          url = "http://www.propublica.org"
          doc = Nokogiri::HTML(self._get_page(url, true))
          doc.css("section#river section h1 a").to_a.map{|l| + l["href"] } #map the Nokogiri link elements to a string representation of a URL.
        end
      end

      ProPublicaScraper.new.scrape do |article_string|
        puts "here is the full text of the ProPublica article: \n #{article_string}"
        #or, do other stuff here.
      end


Why "Upton"
----------------------
<NAME> was a pioneering, muckraking journalist who is most famous for _The Jungle_, a novel portraying the reality of immigrant labor struggles in Chicago meatpacking plants at the start of the 1900s. Upton, the gem, sprang out of a ProPublica project pertaining to labor issues.

License (MIT)
{'fraction_non_alphanumeric': 0.08594249201277955, 'fraction_numerical': 0.002875399361022364, 'mean_word_length': 3.393684210526316, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3326317', 'n_tokens_mistral': 1743, 'n_tokens_neox': 1643, 'n_words': 878}