<commit_before># DocParser

Rubygem: [docparser](http://rubygems.org/gems/docparser)
Docs: http://rubydoc.info/github/jurriaan/docparser/

DocParser is a web scraping/screen scraping tool.
You can use it to easily scrape web sites.

## Features

- XPath and CSS support through Nokogiri
- Support for loading of URLs throug open-uri
- Support for parallel processing of the documents
- 6 Output formats:
  * CSV
  * XLSX
  * HTML
  * YAML
  * JSON
  * Screen (for debugging and development)
  * And more! (easy to extend)

## Installation

Add this line to your application's Gemfile:

    gem 'docparser'

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install docparser

## Usage

See example.rb

## Todo

- Tests

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

## Contributors

- [<NAME>](https://github.com/jurriaan)<commit_msg>Readme update<commit_after># DocParser

DocParser is a web scraping/screen scraping tool.

You can use it to easily scrape web sites.

The gem is called [docparser](http://rubygems.org/gems/docparser).
You can find the documentation [here](http://rubydoc.info/github/jurriaan/docparser/).

## Features

- XPath and CSS support through Nokogiri
- Support for loading of URLs throug open-uri
- Support for parallel processing of the documents
- 6 Output formats:
  * CSV
  * XLSX
  * HTML
  * YAML
  * JSON
  * Screen (for debugging and development)
  * And more! (easy to extend)

## Installation

Add this line to your application's Gemfile:

    gem 'docparser'

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install docparser

## Usage

See example.rb

## Todo

- Tests
- Better examples

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

## Contributors

- [<NAME>](https://github.com/jurriaan)
{'fraction_non_alphanumeric': 0.09865678554886521, 'fraction_numerical': 0.00555812876331635, 'mean_word_length': 3.53781512605042, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '32611', 'n_tokens_mistral': 719, 'n_tokens_neox': 668, 'n_words': 263}