<commit_before># twingly-url-normalizer

[![Build Status](https://magnum.travis-ci.com/twingly/twingly-url-normalizer.png?token=ADz8fWxRD<PASSWORD>Z<PASSWORD>&branch=master)](https://magnum.travis-ci.com/twingly/twingly-url-normalizer)

Ruby gem for URL normalization

## Example

```
irb(main):001:0> Twingly::URL::Normalizer.normalize('http://duh.se')
=> ["http://www.duh.se"]
irb(main):002:0> Twingly::URL::Normalizer.normalize('http://duh.se http://blog.twingly.com/')
=> ["http://www.duh.se", "http://blog.twingly.com/"]
irb(main):003:0>
```

## Tests

Run tests with

    bundle exec rake

### Profiling

You can get some profiling by running

    bundle exec rake test:profile

Note that this isn't a benchmark, we're using [ruby-prof] which will slow things down.

[ruby-prof]: http://ruby-prof.rubyforge.org/<commit_msg>Update example run<commit_after># twingly-url-normalizer

[![Build Status](https://magnum.travis-ci.com/twingly/twingly-url-normalizer.png?token=ADz<PASSWORD>&branch=master)](https://magnum.travis-ci.com/twingly/twingly-url-normalizer)

Ruby gem for URL normalization

## Example

```
[1] pry(main)> require 'twingly-url-normalizer'
=> true
[2] pry(main)> Twingly::URL::Normalizer.normalize('http://duh.se')
=> ["http://www.duh.se/"]
[3] pry(main)> Twingly::URL::Normalizer.normalize('http://duh.se http://blog.twingly.com/')
=> ["http://www.duh.se/", "http://blog.twingly.com/"]
[4] pry(main)> Twingly::URL::Normalizer.normalize('no URL')
=> []
```

## Tests

Run tests with

    bundle exec rake

### Profiling

You can get some profiling by running

    bundle exec rake test:profile

Note that this isn't a benchmark, we're using [ruby-prof] which will slow things down.

[ruby-prof]: http://ruby-prof.rubyforge.org/
{'fraction_non_alphanumeric': 0.18982275586049172, 'fraction_numerical': 0.009719839908519153, 'mean_word_length': 4.972696245733788, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 19, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3760781', 'n_tokens_mistral': 671, 'n_tokens_neox': 632, 'n_words': 119}