<commit_before># NiceCrawler

A simple and nice web crawler.

## Browser

It's Sinatra so it's easy.

```bash
  cd YOUR_PATH/nicecrawler
  bundle install
  sinatra app.rb
```

Then go to http://localhost:3000, input a domain into the text field and
receive your site map.


## Command line

```bash
  cd YOUR_PATH/nicecrawler
  bundle install
  ./bin/nicecrawler http://digitalocean.com > sitemap.json
```

## Ruby scripting

```ruby
  require 'nice_crawler'

  url = 'http://digitalocean.com'

  crawler = NiceCrawler.new(url)
  crawler.crawl
  # or
  crawler.be_nice_and_crawl
```
<commit_msg>Improving readme<commit_after># NiceCrawler

A simple and nice web crawler.

It uses MongoDB to store the sitemap, so you need it to run the script.

http://docs.mongodb.org/manual/installation/

## Command-line

```bash
  cd YOUR_PATH/nicecrawler
  bundle install
  start your mongodb ...

  ./bin/nicecrawler http://digitalocean.com > sitemap.json
```

## Ruby

```ruby
  require 'nice_crawler'

  url = 'http://digitalocean.com'

  crawler = NiceCrawler.new(url)
  crawler.crawl
  puts crawler.sitemap
```

## MongoDB Options

By default it points to **localhost:27017**, but it can be changed passing
additional parameters to the class.

```ruby
  opts = {
    address: 172.16.58.3,
    port: 1122,
    database: 'myownname'
  }

  url = 'http://guivinicius.com'
  NiceCrawler.new(url, opts).crawl
```

{'fraction_non_alphanumeric': 0.11848679514632406, 'fraction_numerical': 0.014989293361884369, 'mean_word_length': 3.551948051948052, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '550371', 'n_tokens_mistral': 560, 'n_tokens_neox': 506, 'n_words': 138}