<commit_before>## Version 3.3.0 (August 16, 2014)

Heuristic changes:

- Certain symbols are marked as "ending punctuation" that may naturally occur
  after letters. When they follow an accented capital letter and look like
  mojibake, they will not be "fixed" without further evidence.
  An example is that "MARQUÉ…" will become "MARQUÉ...", and not "MARQUɅ".

New features:

- `ftfy.explain_unicode` is a diagnostic function that shows you what's going
  on in a Unicode string. It shows you a table with each code point in
  hexadecimal, its glyph, its name, and its Unicode category.

- `ftfy.fixes.decode_escapes` adds a feature missing from the standard library:
  it lets you decode a Unicode string with backslashed escape sequences in it
  (such as "\u2014") the same way that Python itself would.

- `ftfy.streamtester` is a release of the code that I use to test ftfy on
  an endless stream of real-world data from Twitter. With the new heuristics,
  the false positive rate of ftfy is about 1 per 6 million tweets. (See
  the "Accuracy" section of the documentation.)

Deprecations:

- Python 2.6 is no longer supported.

- `remove_unsafe_private_use` is no longer needed in any current version of
  Python. This fixer will disappear in a later version of ftfy.

## Version 3.2.0 (June 27, 2014)

- `fix_line_breaks` fixes three additional characters that are considered line
  breaks in some environments, such as Javascript, and Python's "codecs"
  library. These are all now replaced with \n:
  
    U+0085   <control>, with alias "NEXT LINE"
    U+2028   LINE SEPARATOR
    U+2029   PARAGRAPH SEPARATOR

## Version 3.1.3 (May 15, 2014)

- Fix `utf-8-variants` so it never outputs surrogate codepoints, even on
  Python 2 where that would otherwise be possible.

## Version 3.1.2 (January 29, 2014)

- Fix bug in 3.1.1 where strings with backslashes in them could never be fixed

## Version 3.1.1 (January 29, 2014)<commit_msg>okay, markdown, this is supposed to be a literal<commit_after>## Version 3.3.0 (August 16, 2014)

Heuristic changes:

- Certain symbols are marked as "ending punctuation" that may naturally occur
  after letters. When they follow an accented capital letter and look like
  mojibake, they will not be "fixed" without further evidence.
  An example is that "MARQUÉ…" will become "MARQUÉ...", and not "MARQUɅ".

New features:

- `ftfy.explain_unicode` is a diagnostic function that shows you what's going
  on in a Unicode string. It shows you a table with each code point in
  hexadecimal, its glyph, its name, and its Unicode category.

- `ftfy.fixes.decode_escapes` adds a feature missing from the standard library:
  it lets you decode a Unicode string with backslashed escape sequences in it
  (such as "\u2014") the same way that Python itself would.

- `ftfy.streamtester` is a release of the code that I use to test ftfy on
  an endless stream of real-world data from Twitter. With the new heuristics,
  the false positive rate of ftfy is about 1 per 6 million tweets. (See
  the "Accuracy" section of the documentation.)

Deprecations:

- Python 2.6 is no longer supported.

- `remove_unsafe_private_use` is no longer needed in any current version of
  Python. This fixer will disappear in a later version of ftfy.

## Version 3.2.0 (June 27, 2014)

- `fix_line_breaks` fixes three additional characters that are considered line
  breaks in some environments, such as Javascript, and Python's "codecs"
  library. These are all now replaced with \n:
  
      U+0085   <control>, with alias "NEXT LINE"
      U+2028   LINE SEPARATOR
      U+2029   PARAGRAPH SEPARATOR

## Version 3.1.3 (May 15, 2014)

- Fix `utf-8-variants` so it never outputs surrogate codepoints, even on
  Python 2 where that would otherwise be possible.

## Version 3.1.2 (January 29, 2014)

- Fix bug in 3.1.1 where strings with backslashes in them could never be fixed

## Version 3.1.1 (January 29, 2014)
{'fraction_non_alphanumeric': 0.06828025477707006, 'fraction_numerical': 0.035668789808917196, 'mean_word_length': 3.7130852340936373, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1140153', 'n_tokens_mistral': 1307, 'n_tokens_neox': 1159, 'n_words': 575}