<jupyter_start><jupyter_text>Digital representation of dataReal world data such as pictures and music in the analog world can be represented digitally for computers to use and process. In this tutorial you will see how data can be represented in a variety of commonly used formats in computer science and programming. You will also learn about some basic Python functions designed to manipulate these representations, as well as a specialized library called BitVector. How to run this (and other) Jupyter Notebooks?You can start by clicking "Run" in the top toolbar to step through and execute each block of text and code. While the text blocks will not have any outputs, the code blocks will very likely display an output right below the block. To run any block out of sequence, first click anywhere in the block, then click Run at the top toolbar, or simply hit Ctrl+Enter. Decimal, binary and hex representation of numbersComputers use the binary system, usually consisting of two symbols `1` and `0`, to represent data. This is nothing but the base-2 system, just like we use the base-10 system in everyday life. Internally, these two symbols are mapped to physical quantities such as two voltage levels +5 volts/-5 volts, or the presence/absence of a magnetic field, in order for a computer to process the data. Let us see how binary representation works, for the decimal number 5. In python, to get the binary representation of the number 5, use the `bin` function:<jupyter_code>bin(5)<jupyter_output><empty_output><jupyter_text>We see that the ouput is `0b101`. The first two digits, `0` and lowercase `b` indicate that what follows is to be read in base-2 notation. The next three digits, `101` are the base-2 representation of the decimal number 5.However, for large numbers, the binary representation can be very long and hard to manage. For example:<jupyter_code>bin(65535)<jupyter_output><empty_output><jupyter_text>This is why we use the *hexadecimal* (base-16) representation to help manage large numbers. In hex, we use 16 symbols: `0 1 2 3 4 5 6 7 8 9 A B C D E F`, just like the decimal system uses ten symbols (0-10) and the binary system uses two (0 and 1). The same number is compactly represented in hex using just four hex digits:<jupyter_code>hex(65535)<jupyter_output><empty_output><jupyter_text>We see that the hex representation starts with `0x`, followed by four symbols in the base-16 system: `ffff`. It is very important to keep an eye on the prefix, since the same set of symbols can represent different numbers in different systems. For example, take the set of digits `1000` which we typically read as one thousand in the decimal system. However, their value is different in binary and hex systems!<jupyter_code>int(0b1000)
int(0x1000)<jupyter_output><empty_output><jupyter_text>In fact `1000` in base-2 is `8` in decimal, and `4096` in base-16! This is why it is important to write down the prefix; when there is no prefix it is automatically assumed to be the decimal representation.We can convert between the hex and binary systems easily. Since 16 is a power of 2 (2^4=16), each hex digit can be represented using four bits. Let's see how:<jupyter_code>for i in range(16):
    print("Decimal ", i, " is ", hex(i), "in hex and ", bin(i), "in binary")<jupyter_output><empty_output><jupyter_text>We see that `0xf` is `0b1111` in binary, which is 4 bits long. Since one byte is eight bits long, we can fit two hex digits in a 8 bits - all we have to do is write them side by side! This trick does not work in decimal though, so beware.<jupyter_code>for i in {0x8, 0xb, 0x8b}:
    print("Hex ", hex(i), "is ", bin(i), "in binary")<jupyter_output><empty_output><jupyter_text>We see that `0x8b` is simply the concatenation of the representation of `0x8` and `0xb`! ASCII representation of alphabets and other symbolsWe know how to represent numbers in binary and hex systems. But what about letters such as `k` and symbols such as `&`? How does a computer represent those internally? The answer is through the use of an *encoding scheme* such as ASCII (American Standard Code for Information Interchange) which uses nearly one byte (seven bits) to represent each symbol in the ASCII table. Here's the table for reference:![ASCII table](1000px-ASCII-Table.svg.png)Recently, other encoding schemes such as Unicode have become more common. The UTF-8 Unicode format extends ASCII to use one to four bytes to represent many more symbols. We can represent emojis - 😂 has a hex representation of `0xF09F9882`! That's eight hex digits, which equals four bytes.How do we convert a word or a sentence into ASCII? Simple. We just convert each letter or character to ASCII, and simply concatenate them. Note that a space ` ` and punctuation such as `;` have their own ASCII representation.<jupyter_code>my_str = 'cat;'
for i in my_str:
    print("ASCII representation of ", i, " is ", hex(ord(i)))
print("ASCII representation of ", my_str, " is ", ' '.join(hex(ord(c)) for c in my_str))<jupyter_output><empty_output><jupyter_text>The BitVector classFor convenience, we will use the BitVector class to manipulate ASCII, binary and hex representation. This will be very useful in other exercises such as the notebook on the One Time Pad (OTP) cipher. Let's take an ASCII text string and print its encoding in binary:<jupyter_code>import BitVector

bv = BitVector.BitVector(textstring="cat;")
print(bv) # prints in binary without prefix<jupyter_output><empty_output><jupyter_text>Instead of binary, we can also print it as a string of hex digits (`0` through `f`), which is much more compact, and preferred in most programming environments.<jupyter_code>print(bv.get_hex_string_from_bitvector()) # prints in hex without prefix<jupyter_output><empty_output><jupyter_text>Frequently, you may need to visualize a hex string. Don't worry! BitVector allows you to convert hex strings to ASCII very easily:<jupyter_code>bv1 = BitVector.BitVector(hexstring="6361743b")
print(bv1.get_bitvector_in_ascii())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06753848700546267, 'fraction_numerical': 0.02118854494289025, 'mean_word_length': 5.190573770491803, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '452752', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1755, 'n_tokens_neox': 1625, 'n_words': 879, 'path': 'stage4/06-jupyter/files/01-encoding.ipynb', 'repo_name': 'ADMETE-OHIO/pi-gen', 'size': '6041'}