<jupyter_start><jupyter_text>Mojibake! What the hâ€”ck happened to this text?<NAME>  Mojibake is when text ends up with the wrong Unicode characters due to an encoding mistake.- It's Japanese for "ghost characters"- ftfy is my Python library that fixes them. (get it with `pip install ftfy`)<jupyter_code>ftfy.fix_text("merci de t‚Äö√†√∂¬¨¬©l‚Äö√†√∂¬¨¬©charger le plug-in")<jupyter_output><empty_output><jupyter_text>The mascot of ftfy<jupyter_code>ftfy.fix_text("(Ã\xa0Â¸â€¡'ÃŒâ‚¬Ã¢Å’Â£'ÃŒÂ\x81)Ã\xa0Â¸â€¡")<jupyter_output><empty_output><jupyter_text>This little guy, and in fact every example in this talk, comes from mojibake I found in the wild -- usually on Twitter or in the OSCAR Web Corpus.> **Side note**: there are a lot of tangents I would like to go off on, but this is a 10-minute talk and there's no time for tangents. So I'll be sprinkling in these side notes that I'll be skipping over as I present the talk.>> If you're interested in them, I suggest pausing the video, or reading the Jupyter Notebook version of this talk that I'll be linking later. It used to be that 1 byte = 1 character, so there were at most 256 possible characters of text that could be shown on a computer. Here's some quick code to see them:<jupyter_code>import blessings
term = blessings.Terminal()  # enable colorful text

def displayable_codepoint(codepoint, encoding):
    char = bytes([codepoint]).decode(encoding, 'replace')
    if char == '�':
        return '▓▓'
    elif not char.isprintable():
        return '░░'
    else:
        return char

def show_char_table(encoding):
    print(f"encoding: {encoding}\n     0 1 2 3 4 5 6 7 8 9 a b c d e f\n")
    for row in range(16):
        print(f"{row*16:>02x}", end="   ")
        if row == 0:
            print(ftfy.formatting.display_center(term.green(" control characters "), 32, "░"))
        elif row == 8 and encoding == 'latin-1':
            print(ftfy.formatting.display_center(term.green(" here be dragons "), 32, "░"))
        else:
            for col in range(16):
                char = displayable_codepoint(row * 16 + col, encoding)
                print(f"{char:<2}", end="")
            print()<jupyter_output><empty_output><jupyter_text>ASCII- In the '60s, we agreed that 128 of these bytes should have well-defined meanings as characters of text. That's ASCII- It worked pretty well for monolingual Americans<jupyter_code>show_char_table("ascii")<jupyter_output>encoding: ascii
     0 1 2 3 4 5 6 7 8 9 a b c d e f

00   ░░░░░░[32m control characters [m░░░░░░
10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
20     ! " # $ % & ' ( ) * + , - . / 
30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? 
40   @ A B C D E F G H I J K L M N O 
50   P Q R S T U V W X Y Z [ \ ] ^ _ 
60   ` a b c d e f g h i j k l m n o 
70   p q r s t u v w x y z { | } ~ ░░
80   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
90   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
a0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
b0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
c0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
d0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
e0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
f0   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓<jupyter_text>- Basically everyone decided the other 128 bytes should be characters too- So people started talking about "extended ASCII" which means "whatever my computer does with the other bytes"- A different computer, in a different country or with a different OS, would do something different- This is how mojibake startedLet's take a look at the different ways that ASCII got "extended", which are now called _codepages_. Here's Latin-1. This encoding was used on a lot of UNIX-like systems before they switched to UTF-8, but probably the biggest reason you see it is that it's the first 256 characters of Unicode.If you don't know about decoding text and you just replace each byte with the Unicode character with the same number, Latin-1 is the encoding you get by accident.<jupyter_code>show_char_table('latin-1')<jupyter_output>encoding: latin-1
     0 1 2 3 4 5 6 7 8 9 a b c d e f

00   ░░░░░░[32m control characters [m░░░░░░
10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
20     ! " # $ % & ' ( ) * + , - . / 
30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? 
40   @ A B C D E F G H I J K L M N O 
50   P Q R S T U V W X Y Z [ \ ] ^ _ 
60   ` a b c d e f g h i j k l m n o 
70   p q r s t u v w x y z { | } ~ ░░
80   ░░░░░░░[32m here be dragons [m░░░░░░░░
90   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
a0   ░░¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ░░® ¯ 
b0   ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿ 
c0   À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï 
d0   Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß 
e0   à á â ã ä å æ ç è é ê ë ì í î ï 
f0   ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ<jupyter_text>Dragons? 🐉I labeled rows 80 and 90 as "here be dragons" because they're full of control characters, but nobody ever agreed on what they do.Text that includes these codepoints can be shown as nearly anything, including little boxes, characters from the Windows-1252 character set, or just messing up your whole terminal window.> Side note: Control character 85 was an attempt to resolve the war between Unix line breaks (`0a`), Windows line breaks (`0d 0a`), and Mac Classic line breaks (`0d`), by introducing an _entirely new_ line break that wasn't even in ASCII.>> This is hilarious in retrospect. Clearly nobody would ever repeat that terrible idea, except the Unicode Consortium, who did it twice more.>> Anyway, ftfy has a function that turns all of these into byte `0a`.<jupyter_code>show_char_table('windows-1252')
show_char_table('cp437')
show_char_table('macroman')
show_char_table("windows-1251")<jupyter_output>encoding: windows-1251
     0 1 2 3 4 5 6 7 8 9 a b c d e f

00   ░░░░░░[32m control characters [m░░░░░░
10   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
20     ! " # $ % & ' ( ) * + , - . / 
30   0 1 2 3 4 5 6 7 8 9 : ; < = > ? 
40   @ A B C D E F G H I J K L M N O 
50   P Q R S T U V W X Y Z [ \ ] ^ _ 
60   ` a b c d e f g h i j k l m n o 
70   p q r s t u v w x y z { | } ~ ░░
80   Ђ Ѓ ‚ ѓ „ … † ‡ € ‰ Љ ‹ Њ Ќ Ћ Џ 
90   ђ ‘ ’ “ ” • – — ▓▓™ љ › њ ќ ћ џ 
a0   ░░Ў ў Ј ¤ Ґ ¦ § Ё © Є « ¬ ░░® Ї 
b0   ° ± І і ґ µ ¶ · ё № є » ј Ѕ ѕ ї 
c0   А Б В Г Д Е Ж З И Й К Л М Н О П 
d0   Р С Т У Ф Х Ц Ч Ш Щ Ъ Ы Ь Э Ю Я 
e0   а б в г д е ж з и й к л м н о п 
f0   р с т у ф х ц ч ш щ ъ ы ь э ю я<jupyter_text>So text wasn't portable between different computers<jupyter_code>phrase = "Plus ça change, plus c’est la même chose"

phrase.encode('windows-1252').decode('macroman')<jupyter_output><empty_output><jupyter_text>Vintage '90s mojibakeMaybe you've seen a ".NFO" file like this, telling you about the nice people who provided you with a DOS game for free:<jupyter_code>crack_nfo = r"""
  ───── ────────────── ───────────────── ────────────── ─────────────── ────
 ▄█████▄ ▀█████▄ ████▄ ▀█████████ ████▄   ▄█████▄▀████▄ ▀██████▄ ▀████▄  ▄██▄
 ████████ █████▀ ██████ ▀████████ ██████ ▀███████▌██████  ███████  ████▄█████
 ███ ▀███▌█▀ ▄▄▄█▌▀█████ ▌████ ▄▄▄█▀▐████ ███▀▌▀█▌█▌ ███▌ ██▀▐████ ██████████
 ███  ▐██▌▌  ████  ▌████  ████ ████  ████ ██▌  ▌▐▌█▌ ████ ██  ████ ██▌▐█▌▐███
 ███ ▄███▌▄▄ ████   ████  ████ ████ ▄████ ██▌  █▄▄█▌ ████ ██ ▄████ ██ ▐█▌ ███
 ████████ ██ ████   ████  ████ ████ █████ ██▌  ▀▀██▌▐███▀ ██▐█████ ██  █  ███
 ██████▀ ▄▀▀ ████   ████  ████ ████ ▀████ ██▌███▄▐██▄▀▀ ▄███ ▀████ ██  ▄  ███
 ███▀ ▄▄██   ████  ▐████  ████ ████  ████ ██▌▐▐██▌█▀██▄ ████  ████ ██     ███
 ███  █████▄  ▀▀█  ████▀ ▐████ ░███ ▐████ ███▄▐██▌█▌▐██▌▐███ ▐███░ ██▌   r███
 ░██  █████████▄   ▐██▌▄██████ ▒░██ █████ ███████▌█▌▐███ ███ ███░▒ ███   o██░
 ▒░█   ▀███████▀  ▀ ███▐████▀  ▓▒░█ ▐███▌ ▀██████▐█▌▐███ ███ ▐█░▒▓ ██▌   y█░▒
 - ▌─────▐▀─ ▄▄▄█ ── ▀▀ ───── ────── ▀▀▀ ─ ▐▀▀▀▀ ▀▀ ████ ──── ▀█▀ ─ ▀ ────▐ ─

   ╓────────────────────────[ RELEASE INFORMATION ]───────────────────────╖
╓────────────────────────────────────────────────────────────────────────────╖
║  -/\- THE EVEN MORE INCREDIBLE MACHINE FOR *DOS* FROM SIERRA/DYNAMIX -/\-  ║
╙────────────────────────────────────────────────────────────────────────────╜
"""<jupyter_output><empty_output><jupyter_text>> **Side note**: It's funny that most of the things we call "ASCII art" aren't ASCII.<jupyter_code>print(crack_nfo.encode('cp437').decode('windows-1252'))<jupyter_output>ÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄ
 ÜÛÛÛÛÛÜ ßÛÛÛÛÛÜ ÛÛÛÛÜ ßÛÛÛÛÛÛÛÛÛ ÛÛÛÛÜ   ÜÛÛÛÛÛÜßÛÛÛÛÜ ßÛÛÛÛÛÛÜ ßÛÛÛÛÜ  ÜÛÛÜ
 ÛÛÛÛÛÛÛÛ ÛÛÛÛÛß ÛÛÛÛÛÛ ßÛÛÛÛÛÛÛÛ ÛÛÛÛÛÛ ßÛÛÛÛÛÛÛÝÛÛÛÛÛÛ  ÛÛÛÛÛÛÛ  ÛÛÛÛÜÛÛÛÛÛ
 ÛÛÛ ßÛÛÛÝÛß ÜÜÜÛÝßÛÛÛÛÛ ÝÛÛÛÛ ÜÜÜÛßÞÛÛÛÛ ÛÛÛßÝßÛÝÛÝ ÛÛÛÝ ÛÛßÞÛÛÛÛ ÛÛÛÛÛÛÛÛÛÛ
 ÛÛÛ  ÞÛÛÝÝ  ÛÛÛÛ  ÝÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ  ÛÛÛÛ ÛÛÝ  ÝÞÝÛÝ ÛÛÛÛ ÛÛ  ÛÛÛÛ ÛÛÝÞÛÝÞÛÛÛ
 ÛÛÛ ÜÛÛÛÝÜÜ ÛÛÛÛ   ÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ ÜÛÛÛÛ ÛÛÝ  ÛÜÜÛÝ ÛÛÛÛ ÛÛ ÜÛÛÛÛ ÛÛ ÞÛÝ ÛÛÛ
 ÛÛÛÛÛÛÛÛ ÛÛ ÛÛÛÛ   ÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ ÛÛÛÛÛ ÛÛÝ  ßßÛÛÝÞÛÛÛß ÛÛÞÛÛÛÛÛ ÛÛ  Û  ÛÛÛ
 ÛÛÛÛÛÛß Üßß ÛÛÛÛ   ÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ ßÛÛÛÛ ÛÛÝÛÛÛÜÞÛÛÜßß ÜÛÛÛ ßÛÛÛÛ ÛÛ  Ü  ÛÛÛ
 ÛÛÛß ÜÜÛÛ   ÛÛÛÛ  ÞÛÛÛÛ  ÛÛÛÛ ÛÛÛÛ  ÛÛÛÛ ÛÛÝÞÞÛÛÝÛßÛÛÜ ÛÛÛÛ  ÛÛÛÛ ÛÛ     ÛÛÛ
 ÛÛÛ  ÛÛÛÛÛÜ  ßßÛ  ÛÛÛÛß ÞÛÛÛÛ °ÛÛÛ ÞÛÛÛÛ ÛÛÛÜÞÛÛÝÛÝÞÛÛÝÞÛÛÛ ÞÛÛÛ° ÛÛÝ   rÛÛÛ
 °ÛÛ  ÛÛÛÛÛÛÛÛÛÜ   ÞÛÛÝÜÛÛÛÛÛÛ ±°ÛÛ ÛÛÛÛÛ ÛÛÛÛÛÛÛÝÛÝÞÛÛÛ ÛÛÛ ÛÛÛ°± ÛÛÛ   oÛÛ°
 ±°Û   ßÛÛÛÛÛÛÛß  ß ÛÛÛÞÛÛÛÛß  ²±°Û ÞÛÛÛÝ ßÛÛÛÛÛÛÞÛÝÞÛÛÛ ÛÛÛ ÞÛ°±² ÛÛÝ   yÛ°±
 - ÝÄÄÄÄÄÞßÄ ÜÜÜÛ ÄÄ ßß ÄÄÄÄÄ ÄÄÄÄÄÄ ßßß Ä Þßßßß ßß ÛÛÛÛ ÄÄÄÄ ßÛ[...]<jupyter_text>Here comes UTF-8Instead of using the other 128 bytes for 128 more characters, what if we used them as a variable-length encoding for the whole rest of Unicode?There were other variable-length encodings, but UTF-8 is well-designed.- It leaves ASCII as ASCII- It doesn't overlap with ASCII _ever_: ASCII bytes always exclusively stand for the characters you expect- It's self-synchronizing, so you can tell where each character starts and ends even with no contextEveryone recognized how good this idea was and switched every system to UTF-8. Encoding problems were solved forever, the end.I'm kidding. UTF-8 is great but not everyone adopted it, especially not Microsoft who had just done _all that work_ to switch Windows APIs to UTF-16, which nobody likes. So now we have more kinds of mojibake.<jupyter_code># Code to look at the encoding of each character in UTF-8

def show_utf8(text):
    for char in text:
        char_bytes = char.encode('utf-8')
        byte_sequence = ' '.join([f"{byte:>02x}" for byte in char_bytes])
        print(f"{char} = {byte_sequence}")

text = "l’Hôpital"
show_utf8(text)<jupyter_output>l = 6c
’ = e2 80 99
H = 48
ô = c3 b4
p = 70
i = 69
t = 74
a = 61
l = 6c<jupyter_text>What happens when not everyone is on board with UTF-8<jupyter_code>text.encode('utf-8')
print(text.encode('utf-8').decode('windows-1252'))
print(text.encode('utf-8').decode('windows-1252').encode('utf-8').decode('windows-1252'))
EXAMPLES = [
    "Merci de t‚Äö√†√∂¬¨¬©l‚Äö√†√∂¬¨¬©charger le plug-in",
    
    "The Mona Lisa doesnÃƒÂ¢Ã¢â€šÂ¬Ã¢â€žÂ¢t have eyebrows.",
    
    "I just figured out how to tweet emojis! â\x9a½í\xa0½í¸\x80í\xa0½í¸\x81í\xa0½í¸"
    "\x82í\xa0½í¸\x86í\xa0½í¸\x8eí\xa0½í¸\x8eí\xa0½í¸\x8eí\xa0½í¸\x8e",
]<jupyter_output><empty_output><jupyter_text>We can recognize UTF-8 mojibake by its distinct patternsThe pattern of bytes that makes UTF-8 self-synchronizing is a pattern we can recognize even via a different encoding.The example `doesnÃƒÂ¢Ã¢â€šÂ¬Ã¢â€žÂ¢t` is recognizable as UTF-8 / Windows-1252 mojibake, for example. `t‚Äö√†√∂¬¨¬©l‚Äö√†√∂¬¨¬©charger` is recognizable as UTF-8 / MacRoman mojibake.When we see such a pattern, we encode as the appropriate other encoding, then decode as UTF-8.<jupyter_code>EXAMPLES[0].encode('macroman').decode('utf-8')
_.encode('macroman').decode('utf-8')
_.encode('macroman').decode('utf-8')<jupyter_output><empty_output><jupyter_text>This is a job that the computer can do for us<jupyter_code># Some code to format the output of ftfy

from ftfy import fix_and_explain
from pprint import pprint

def show_explanation(text):
    print(f"Original: {text}")
    fixed, expl = fix_and_explain(text)
    print(f"   Fixed: {fixed}\n")
    pprint(expl)
show_explanation(EXAMPLES[0])
show_explanation(EXAMPLES[1])
show_explanation(EXAMPLES[2])<jupyter_output>Original: I just figured out how to tweet emojis! â½í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸í ½í¸
   Fixed: I just figured out how to tweet emojis! ⚽😀😁😂😆😎😎😎😎

[('encode', 'latin-1'), ('decode', 'utf-8-variants')]<jupyter_text>> **Side note**: ftfy adds encodings to Python like `sloppy-windows-1252` and `utf-8-variants` when it's imported. Python is very strict about encoding standards, but to deal with real mojibake, we have to be very loose about them. For example, this tweet requires us to recognize and simulate a broken implementation of UTF-8.  Avoiding false positivesftfy only changes text that trips its mojibake-detector regex.Here are some examples that ftfy _could_ consider to be UTF-8 mojibake and try to "fix", but thankfully it doesn't:<jupyter_code>NEGATIVE_EXAMPLES = [
    "Con il corpo e lo spirito ammaccato,\u00a0è come se nel cuore avessi un vetro conficcato.",
    "2012—∞",
    "TEM QUE SEGUIR, SDV SÓ…",
    "Join ZZAJÉ’s Official Fan List",
    "(-1/2)! = √π",
    "OK??:(   `¬´    ):"
]

for example in NEGATIVE_EXAMPLES:
    # ftfy doesn't "fix" these because they're not broken, but we can manually try fixes
    try:
        print(example.encode('sloppy-windows-1252').decode('utf-8'))
    except UnicodeError:
        print(example.encode('macroman').decode('utf-8'))
    assert ftfy.fix_encoding(example) == example<jupyter_output>Con il corpo e lo spirito ammaccato,ʏ come se nel cuore avessi un vetro conficcato.
2012Ѱ
TEM QUE SEGUIR, SDV SӅ
Join ZZAJɒs Official Fan List
(-1/2)! = ù
OK??:(   `«    ):<jupyter_text>The "badness" metricftfy doesn't just look for UTF-8-like patterns of characters, it also makes sure they are unlikely to be the intended text.- Improbable combinations: accented letters next to currency signs, math symbols next to console line art- Lots of things involving capital Ã and Â, where it doesn't look like they're being used for real in a capitalized word- Unloved characters like `¶` PILCROW SIGN, `‡` DOUBLE DAGGER, `◊` LOZENGE next to other mojibake-related characters...and many more cases that it looks for in a big regex.Strings that match the regex can be re-decoded. Specific character sequences that match the regex can be reinterpreted even if they're inconsistent with the rest of the string.> **Side note**: We used to try to categorize every Unicode character to find "badness". Now we only categorize the 400 or so characters that actually can appear in UTF-8 mojibake, because ftfy wouldn't have a reason to replace the other characters anyway.<jupyter_code>text = "Ã perturber la rÃ©flexion des thÃ©ologiens jusqu'Ã nos jours"

# We want to highlight the matches to this regular expression:
ftfy.badness.BADNESS_RE.findall(text)

# We'll just highlight it manually:
term = blessings.Terminal()
highlighted_text = term.on_yellow("Ã ") + "perturber la r" + term.on_yellow("Ã©") + "flexion des th" + term.on_yellow("Ã©") + "ologiens jusqu'Ã nos jours"
# Highlighted text shows matches for the 'badness' expression.
# If we've confirmed from them that this is mojibake, and there's a consistent fix, we
# can fix even text in contexts that were too unclear for the regex, such as the final Ã.

print(highlighted_text)
print(ftfy.fix_text(highlighted_text))<jupyter_output>[43mÃ [mperturber la r[43mÃ©[mflexion des th[43mÃ©[mologiens jusqu'Ã nos jours
à perturber la réflexion des théologiens jusqu'à nos jours
{'fraction_non_alphanumeric': 0.18023633294377353, 'fraction_numerical': 0.024412413972211403, 'mean_word_length': 3.810430980637102, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 14, 'word_list_counts': {'cursed_substrings.json': 43, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '51979', 'license': 'MIT', 'n_tokens_mistral': 7339, 'n_tokens_neox': 7363, 'n_words': 2194, 'path': 'notebook/ftfy talk.ipynb', 'repo_name': 'pombredanne/python-ftfy', 'size': '15407'}