<commit_before>= A UTF-8 Validator State Machine

Provides an implementation of a state machine for validating UTF-8 encoded strings.  Clients may request that encoding errors be reported in several ways:

* simple true / false indicator
* a raised exception

== What This gem does Not Provide

* UTF-8 Encoding
* UTF-8 Decoding

That functionality is left as an exercise for the reader.

== Thanks To

The Unicode Consortium:: At http://unicode.org/ for all the information published there.
<NAME>:: For the state machine algorithm.  See: http://unicode.org/mail-arch/unicode-ml/y2003-m02/att-0467/01-The_Algorithm_to_Valide_an_UTF-8_String
<NAME>:: For invalid test data.  http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

== A Word On Ruby Versions

It is expected that this validator will be used in Ruby environments prior to 1.9.x. However, nothing prohibits it's use with Ruby 1.9.

== Reporting Issues

Please report issues on the tracker at github.

== Contributing to the utf8_validator gem
 
* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
* Fork the project.
* Start a feature/bugfix branch.
* Commit and push until you are happy with your contribution.
* Make sure to add tests for it. This is important so it does not break in in a future version unintentionally.
* Please try not to modify the Rakefile or VERSION file. If you require your own version please isolate the version update to its own commit so cherry-pick or rebase can be used to skip it.
* Request a pull.

== Copyright

Copyright (c) 2011 <NAME>. See LICENSE.txt for
further details.
<commit_msg>Update README with more general information.<commit_after>= A UTF-8 Validator State Machine

Provides an implementation of a state machine for validating UTF-8 encoded strings.  Clients may request that encoding errors be reported in several ways:

* simple true / false indicator
* a raised exception

== What This gem does Not Provide

* UTF-8 Encoding
* UTF-8 Decoding

That functionality is left as an exercise for the reader.

== Thanks To

The Unicode Consortium:: At http://unicode.org/ for all the information published there.
<NAME>:: For the state machine algorithm.  See: http://unicode.org/mail-arch/unicode-ml/y2003-m02/att-0467/01-The_Algorithm_to_Valide_an_UTF-8_String
<NAME>:: For invalid test data.  http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

== Useful Information

Other interesting and/or useful information can be found:

* http://software.hixie.ch/utilities/cgi/unicode-decoder/utf8-decoder
* http://bjoern.hoehrmann.de/utf-8/decoder/dfa/

== A Word On Ruby Versions

It is expected that this validator will be used in Ruby environments prior to 1.9.x. However, nothing prohibits it's use with Ruby 1.9.

== Reporting Issues

Please report issues on the tracker at github:

* https://github.com/gmallard/utf8_validator/issues

== Web Based Documentation

Human readable documentation can be found at:

* http://gmallard.github.com/utf8_validator

== Contributing to the utf8_validator gem
 
* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
* Fork the project.
* Start a feature/bugfix branch.
* Commit and push until you are happy with your contribution.
* Make sure to add tests for it. This is important so it does not break in in a future version unintentionally.
* Please try not to modify the Rakefile or VERSION file. If you require your own version please isolate the version update to its own commit so cherry-pick or rebase can be used to skip it.
* Request a pull.

== Copyright

Copyright (c) 2011 <NAME>. See LICENSE.txt for
further details.

{'fraction_non_alphanumeric': 0.0710452338359315, 'fraction_numerical': 0.015844620495783286, 'mean_word_length': 4.069948186528498, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7002159', 'n_tokens_mistral': 1165, 'n_tokens_neox': 1080, 'n_words': 525}