<commit_before>[![Build Status](https://secure.travis-ci.org/bogrobotten/fetch.png)](http://travis-ci.org/bogrobotten/fetch)

# Fetch

![Fetch](http://i.imgur.com/B8TXlri.png)

Fetch enables easy fetching of data from multiple web sources.
It was extracted from [Bogrobotten](http://www.bogrobotten.dk) where we use it
to fetch prices and other stuff from multiple merchants.
We use it for price comparison, but you can use it for anything that involves
fetching data from external sources.

Fetch uses the [Typhoeus](https://github.com/typhoeus/typhoeus) gem for fast
and reliable asynchronous fetches from multiple URLs.

## Installation

Add this line to your application's *Gemfile*:

```ruby
gem "fetch"
```

Then run:

```bash
$ bundle
```

## Example

In *app/models/user.rb*:

```ruby
class User < ActiveRecord::Base
  def fetcher
    @fetcher ||= UserFetcher.new(self)
  end
end
```

In *app/fetchers/user_fetcher.rb*:

```ruby
class UserFetcher < Fetch::Base
  modules Facebook::UserInfoFetch,
          Github::UserInfoFetch
end
```

In *lib/facebook/user_info_fetch.rb*:

```ruby
module Facebook
  class UserInfoFetch < Fetch::Module
    include Fetch::Simple
    include Fetch::JSON

    url do
      "http://graph.facebook.com/#{fetchable.login}"
    end

    process do |user_info|
      fetchable.update_attribute :facebook_id, user_info["id"]
    end
  end
end
```

In *lib/github/user_info_fetch.rb*

```ruby
module Github
  class UserInfoFetch < Fetch::Module
    include Fetch::JSON

    # Request for user ID
    request do |req|
      req.url = "https://api.github.com/users/#{fetchable.login}"
      req.process do |user|
        fetchable.update_attribute :github_id, user["id"]
      end
    end

    # Request for repos
    request do |req|
      req.url = "https://api.github.com/users/#{fetchable.login}/repos"
      req.process do |repos|
        repo_names = repos.map { |r| r["name"] }
        fetchable.update_attribute :github_repos, repo_names
      end
    end
  end
end
```

Then, when everything is set up, you can do:

```ruby
user = User.find(123)
user.fetcher.fetch
```

This will run three requests – one for Facebook and two for GitHub – and update
the user model with a Facebook user ID, a GitHub user ID, and a list of GitHub
repos.

## Good to know

### Adding defaults to your requests

Each fetch module has a `defaults` callback that you can use to set up defaults
for all requests in that modules.

```ruby
class UserInfoFetch < Fetch::Module
  defaults do |req|
    req.user_agent = "My Awesome Bot!"
  end

  request do |req|
    req.url = "http://test.com"
    req.process do |body|
      # Do some processing
    end
  end
end
```

This will add the user agent `My Awesome Bot!` to all requests in the
`UserInfoFetch` module.

The `defaults` callback is inherited, like all other callbacks, so if you have
a base fetch class that you subclass, the `defaults` callback in the superclass
will be run in all subclasses.

### Parsing JSON

Fetch has a module for automatically parsing the request body as JSON before
it is sent to the process block.

```ruby
class UserInfoFetch < Fetch::Module
  include Fetch::JSON

  request do |req|
    req.url = "http://api.test.com/user"
    req.process do |json|
      # Do something with the JSON.
    end
  end
end
```

### Dynamically loading fetch modules

You can load fetch modules dynamically using the `load` callback. Normally, the
modules defined with `modules` are instantiated directly. When you use the
`load` callback, this will determine how your modules are loaded.

```ruby
class UserFetcher < Fetch::Base
  modules :user_info_fetch, :status_fetch

  load do |modules|<commit_msg>Add info on failures and errors to readme<commit_after>[![Build Status](https://secure.travis-ci.org/bogrobotten/fetch.png)](http://travis-ci.org/bogrobotten/fetch)

# Fetch

![Fetch](http://i.imgur.com/B8TXlri.png)

Fetch enables easy fetching of data from multiple web sources.
It was extracted from [Bogrobotten](http://www.bogrobotten.dk) where we use it
to fetch prices and other stuff from multiple merchants.
We use it for price comparison, but you can use it for anything that involves
fetching data from external sources.

Fetch uses the [Typhoeus](https://github.com/typhoeus/typhoeus) gem for fast
and reliable asynchronous fetches from multiple URLs.

## Installation

Add this line to your application's *Gemfile*:

```ruby
gem "fetch"
```

Then run:

```bash
$ bundle
```

## Example

In *app/models/user.rb*:

```ruby
class User < ActiveRecord::Base
  def fetcher
    @fetcher ||= UserFetcher.new(self)
  end
end
```

In *app/fetchers/user_fetcher.rb*:

```ruby
class UserFetcher < Fetch::Base
  modules Facebook::UserInfoFetch,
          Github::UserInfoFetch
end
```

In *lib/facebook/user_info_fetch.rb*:

```ruby
module Facebook
  class UserInfoFetch < Fetch::Module
    include Fetch::Simple
    include Fetch::JSON

    url do
      "http://graph.facebook.com/#{fetchable.login}"
    end

    process do |user_info|
      fetchable.update_attribute :facebook_id, user_info["id"]
    end
  end
end
```

In *lib/github/user_info_fetch.rb*

```ruby
module Github
  class UserInfoFetch < Fetch::Module
    include Fetch::JSON

    # Request for user ID
    request do |req|
      req.url = "https://api.github.com/users/#{fetchable.login}"
      req.process do |user|
        fetchable.update_attribute :github_id, user["id"]
      end
    end

    # Request for repos
    request do |req|
      req.url = "https://api.github.com/users/#{fetchable.login}/repos"
      req.process do |repos|
        repo_names = repos.map { |r| r["name"] }
        fetchable.update_attribute :github_repos, repo_names
      end
    end
  end
end
```

Then, when everything is set up, you can do:

```ruby
user = User.find(123)
user.fetcher.fetch
```

This will run three requests – one for Facebook and two for GitHub – and update
the user model with a Facebook user ID, a GitHub user ID, and a list of GitHub
repos.

## Good to know

### Adding defaults to your requests

Each fetch module has a `defaults` callback that you can use to set up defaults
for all requests in that modules.

```ruby
class UserInfoFetch < Fetch::Module
  defaults do |req|
    req.user_agent = "My Awesome Bot!"
  end

  request do |req|
    req.url = "http://test.com"
    req.process do |body|
      # Do some processing
    end
  end
end
```

This will add the user agent `My Awesome Bot!` to all requests in the
`UserInfoFetch` module.

The `defaults` callback is inherited, like all other callbacks, so if you have
a base fetch class that you subclass, the `defaults` callback in the superclass
will be run in all subclasses.

### Handling HTTP failures

HTTP failures can be handled using the `failure` callback. If you want to
handle failures for all requests generally, you can use the module-wide
`failure` callback:

```ruby
class UserInfoFetch < Fetch::Module
  request do |req|
    req.url = "http://test.com/something-failing"
    req.process do |body|
      # Do something if successful.
    end
  end

  failure do |code, url|
    Rails.logger.info "Fetching from #{url} failed: #{code}"
  end
end
```

If you want to handle failures on the specific requests instead:

```ruby
class UserInfoFetch < Fetch::Module
  request do |req|
    req.url = "http://test.com/something-failing"
    req.process do |body|
      # Do something if successful.
    end
    req.failure do |code, url|
      # Handle the failure
    end
  end
end
```

When you handle failures directly on the request, the general `failure`
callback isn't called.

**Note:** If you don't specify a `failure` callback at all, HTTP failures are ignored,
and processing skipped for the failed request.

### Handling errors

Sometimes a URL will return something that potentially makes your processing
code fail. To prevent this from breaking your whole fetch, you can handle
errors using the `error` callback:

```ruby
class UserInfoFetch < Fetch::Module
  request do |req|
    req.url = "http://test.com/something-failing"
    req.process do |body|
      # Do something if successful.
    end
  end

  error do |exception|
    Rails.logger.info "An error occured: #{exception.message}\n" +
                      exception.backtrace.join("\n")
    raise exception if ["development", "test"].include?(Rails.env)
  end
end
```

You can also do it directly on the requests:

```ruby
class UserInfoFetch < Fetch::Module
  request do |req|
    req.url = "http://test.com/something-failing"
    req.process do |body|
      # Do something if successful.
    end
    req.error do |exception|
      # Handle the error
    end
  end
end
```

If you handle errors directly on the requests, the general `error` callback
isn't run.

**Note:** If you don't do any error handling in one of the two ways shown
above, any exceptions that occur when processing will be raised, causing the
whole fetch to fail. So please add error handling :blush:

### Parsing JSON

Fetch has a module for automatically parsing the request body as JSON before
it is sent to the process block.

```ruby
class UserInfoFetch < Fetch::Module
  include Fetch::JSON

  request do |req|
    req.url = "http://api.test.com/user"
    req.process do |json|
      # Do something with the JSON.
    end
  end
end
```

### Dynamically loading fetch modules

You can load fetch modules dynamically using the `load` callback. Normally, the
modules defined with `modules` are instantiated directly. When you use the
`load` callback, this will determine how your modules are loaded.

```ruby
class UserFetcher < Fetch::Base
  modules :user_info_fetch, :status_fetch

  load do |modules|
{'fraction_non_alphanumeric': 0.10024803637866887, 'fraction_numerical': 0.0008267879288962381, 'mean_word_length': 3.5009302325581397, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '33126', 'n_tokens_mistral': 3053, 'n_tokens_neox': 2905, 'n_words': 1127}