<jupyter_start><jupyter_text>What are Strings?PyTennessee 2020 Open Space<NAME> | @xvillaneau | <EMAIL>  What is a string? - **Goal:**    Represent text in a computer- **Problem:**    Computers only understand numbers- **Solution:**    Store letters as numbers!  String = Sequence of numbers<jupyter_code>list(b"Hello World!")<jupyter_output><empty_output><jupyter_text>Any structure as long as it's ordered:  array, linked list, tree…  How do strings work?Matching letters to numbers = **Encoding** Binary text encoding is _much_ older than computers.History time. ![alt text](Baudot_Code_-_from_1888_patent.png)  The Baudot Code* Invented 1870 (patented later)* 5-bit **stateful** encoding* Designed for use by humans  The Murray Code* Invented 1901* Better Baudot code for machines* Introduced **control characters**Industry standard up through the 1950s ![Colossus](Colossus.jpg) A Colossus Mark 2 codebreaking omputer, 1943. Operating staff had 272 women and 27 men!  The ASCII Encoding- 7-bit code (128 characters)- Published in 1963- Designed for Teletype / Teleprinters- Base of practically all other encodings created since  The ASCII Encoding<jupyter_code>import numpy as np
import pandas as pd

ascii_chars = [
    "␀␁␂␃␄␅␆␇␈␉␊␋␌␍␎␏",
    "␐␑␒␓␔␕␖␗␘␙␚␛␜␝␞␟",
    " !\"#$%&'()*+,-./",
    "0123456789:;<=>?",
    "@ABCDEFGHIJKLMNO",
    "PQRSTUVWXZY[\\]^_",
    "`abcdefghijklmno",
    "pqrstuvwxyz{|}~␡",
]
ascii_col = [f"_{n:x}" for n in range(16)]
ascii_row = [f"{n:x}_" for n in range(8)]
np_ascii = np.array([list(line) for line in ascii_chars])
ascii_table = pd.DataFrame(np_ascii, columns=ascii_col, index=ascii_row)
ascii_table<jupyter_output><empty_output><jupyter_text>Meanwhile at IBM…* Created the EBCDIC 8-bit encoding* Designed for System/360 released in 1964* Totally ASCII-incompatible System/360 popularized using 8 bits as a byte.  What about other languages?"Extended ASCII" codes use the 8th bit for 128 extra characters. - ISO-8859- KOI8- Windows-1252- Mac OS Roman- …  UnicodeHaving all these encodings is inconvenient, because Internet. **Unicode** = international standard for encoding text.Currently includes 137,994 characters! (more soon)  Unicode* Code points allowed: 0 - 1,114,111 (`0x10FFFF`)* Codes 0 - 255 based on Latin-1* Each 16-bit block is called a _plane_  UTF-8UTF-8 is the most popular way to encode Unicode. - Superset of ASCII- Uses between 1 and 4 bytes per character- Never produces null bytes except for `U+0`  UTF-8| From      | To         | Byte 1     | Byte 2     | Byte 3     | Byte 4     ||-----------|------------|------------|------------|------------|------------||     `U+0` |     `U+7F` | `0xxxxxxx` |            |            |            ||    `U+80` |    `U+7FF` | `110xxxxx` | `10xxxxxx` |            |            ||   `U+800` |   `U+FFFF` | `1110xxxx` | `10xxxxxx` | `10xxxxxx` |            || `U+10000` | `U+10FFFF` | `11110xxx` | `10xxxxxx` | `10xxxxxx` | `10xxxxxx` |  UTF-8 ExampleThe Thumbs Up (👍) emoji: `U+1F44D` In binary: `0001.1111.0100.0100.1101`  Re-arranged: `000.011111.010001.001101` UTF-8: `11110000.10011111.10010001.10001101`  Or: `F0.9F.91.8D`  ⚠ Unicode ≠ UTF-8 ⚠- **Unicode:**    Assigns _code points_ to characters- **UTF-8:**    Converts code points into _bytes_  Strings in Python> Strings are immutable sequences of Unicode code points.https://docs.python.org/3.8/library/stdtypes.htmltextseq  Demonstration  Demonstration (plan B)<jupyter_code>s = "Pandora \U0001F44D"
s
s.encode()
s.encode().decode('cp1252', 'replace')<jupyter_output><empty_output><jupyter_text>A short memory exercise<jupyter_code>s = "ewQ;KDWad qwe24a]s[awv;'15324ansdanx!$0(" * 2500
len(s)
import sys
sys.getsizeof(s)<jupyter_output><empty_output><jupyter_text>A short memory exercise<jupyter_code>s1 = s + 'é'
s2 = s + '€'
s3 = s + '👍'
sys.getsizeof(s1)
sys.getsizeof(s2)
sys.getsizeof(s3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15354938271604937, 'fraction_numerical': 0.0684156378600823, 'mean_word_length': 3.9227848101265823, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '379148', 'license': 'MIT', 'n_tokens_mistral': 1660, 'n_tokens_neox': 1411, 'n_words': 391, 'path': 'intro_to_strings/What_are_strings_PyTN2020.ipynb', 'repo_name': 'xvillaneau/talks', 'size': '3911'}