<commit_before># EncodingEstimator: Detect encoding of strings

![Build Status](https://git.iftrue.de/okirmis/encoding_estimator/badges/master/build.svg)

This gem allows you to detect the encoding of strings/files based on their content. This can be useful if you need to load data from sources with unknown encodings. The gem uses character distribution statistics to check which encoding is the one that gives you the best results.

## Usage in Ruby Code

The gem has two major high level methods. The first one can be used when you want to know, how a string is encoded:

```ruby
detection = EncodingEstimator.detect( File.read( 'foo.txt' ), languages: [ :en, :de ] )
puts "Encoding: #{detection.result.encoding}"
```

The second one is a shortcut you can use in case you just want to be sure to get a string of an unknown encoding as a UTF-8 encoded string (should be the ruby default):

```ruby<commit_msg>Added coverage badge to README<commit_after># EncodingEstimator: Detect encoding of strings

![Build Status](https://git.iftrue.de/okirmis/encoding_estimator/badges/master/build.svg)
![Code Covergae](https://git.iftrue.de/okirmis/encoding_estimator/badges/master/coverage.svg)

This gem allows you to detect the encoding of strings/files based on their content. This can be useful if you need to load data from sources with unknown encodings. The gem uses character distribution statistics to check which encoding is the one that gives you the best results.

## Usage in Ruby Code

The gem has two major high level methods. The first one can be used when you want to know, how a string is encoded:

```ruby
detection = EncodingEstimator.detect( File.read( 'foo.txt' ), languages: [ :en, :de ] )
puts "Encoding: #{detection.result.encoding}"
```

The second one is a shortcut you can use in case you just want to be sure to get a string of an unknown encoding as a UTF-8 encoded string (should be the ruby default):

```ruby
{'fraction_non_alphanumeric': 0.07855297157622738, 'fraction_numerical': 0.0010335917312661498, 'mean_word_length': 4.579250720461095, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1902218', 'n_tokens_mistral': 547, 'n_tokens_neox': 518, 'n_words': 267}