<commit_before># FeedParser

Rss and Atom feed parser built on top of Nokogiri. Supports custom sanitizers.

## Install

Add to Gemfile

    gem "feed_parser"

## Usage

    # the most basic use case
    fp = FeedParser.new(:url => "http://example.com/feed/")
    # with sanitizer
    fp = FeedParser.new(:url => "http://example.com/feed/", :sanitizer => MyBestestSanitizer.new)
    # sanitizing custom field set
    fp = FeedParser.new(:url => "http://example.com/feed/", :sanitizer => MyBestestSanitizer.new, :fields_to_sanitize => [:title, :content])
    
    # parse the feed
    feed = fp.parse
    
    # using parsed feed in your code
    feed.as_json
    # => {:title => "Feed title", :url => "http://example.com/feed/", :items => [{:guid => , :title => , :author => ...}]}
    
    feed.items.each do |feed_item|
      pp feed_item
    end

    # you can also pass http options to be used for the connection
    # for available options, check out the OpenURI documentation: http://apidock.com/ruby/OpenURI
    fp = FeedParser.new(:url => "http://example.com/feed/", :http => {:ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE})

## Running tests

Install dependencies by running `bundle install`.

Run rspec tests:

    $ bundle exec rspec

## Contributing

Fork, hack, push, create a pull request.<commit_msg>Updated README.<commit_after># FeedParser

Rss and Atom feed parser built on top of Nokogiri. Supports custom sanitizers.

## Install

Add to Gemfile

    gem "feed_parser"

## Usage

    # the most basic use case
    fp = FeedParser.new(:url => "http://example.com/feed/")
    # with sanitizer
    fp = FeedParser.new(:url => "http://example.com/feed/", :sanitizer => MyBestestSanitizer.new)
    # sanitizing custom field set
    fp = FeedParser.new(:url => "http://example.com/feed/", :sanitizer => MyBestestSanitizer.new, :fields_to_sanitize => [:title, :content])
    
    # parse the feed
    feed = fp.parse
    
    # using parsed feed in your code
    feed.as_json
    # => {:title => "Feed title", :url => "http://example.com/feed/", :items => [{:guid => , :title => , :author => ...}]}
    
    feed.items.each do |feed_item|
      pp feed_item
    end

    # you can also pass http options to be used for the connection
    # for available options, check out the OpenURI documentation: http://apidock.com/ruby/OpenURI
    fp = FeedParser.new(:url => "http://example.com/feed/", :http => {:ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE})

If the fetched XML is not a valid RSS or an ATOM feed, a FeedParser::UnknownFeedType is raised in FeedParser#parse.

## Running tests

Install dependencies by running `bundle install`.

Run rspec tests:

    $ bundle exec rspec

## Contributing

Fork, hack, push, create a pull request.
{'fraction_non_alphanumeric': 0.1353219352491815, 'fraction_numerical': 0.0, 'mean_word_length': 3.2177914110429446, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2417090', 'n_tokens_mistral': 898, 'n_tokens_neox': 864, 'n_words': 269}