<jupyter_start><jupyter_text>Lab1 - crypto - <NAME> <EMAIL> <EMAIL>  Exercise 0: make utilitiesThis exercise is not strictly mandatory, but it will be useful for therest of the lab.Write 6 functions `bin2txt`, `bin2hex`, `txt2bin`, `hex2bin`, `hex2txt`, `txt2hex`, that convert between the following representations:  - hex: `"426f6f6d"` (more precisely, a string to be interepreted as      hexadecimal)  - text: `"Boom"`  - binary: `b"Boom"` in PythonDepending on the language, you may not have to distinguish betweenbinary and text, for instance in C it is the same thing, however inPython one has type `str` whereas the other has type `bytes`.You are not expected to write any complex algorithm here, justdelegate to the correct utility functions of your language if they areprovided. In other words don't rewrite yet another routine tohand-parse hexadecimal.<jupyter_code>def bin2txt(arg: bytes) -> str:
    return arg.decode("utf-8")

def bin2hex(arg: bytes) -> str:
    return arg.hex()

def txt2bin(arg: str) -> bytes:
    return arg.encode()

def hex2bin(arg: str) -> bytes:
    return bytes.fromhex(arg)

def hex2txt(arg: str) -> str:
    return bin2txt(hex2bin(arg))

def txt2hex(arg: str) -> str:
    return bin2hex(txt2bin(arg))<jupyter_output><empty_output><jupyter_text>Exercise 1: encrypt xorWrite a function that encrypts a text with a xor key. The idea issimple: to obtain the first byte of the ciphertext, xor the first byteof the text and the first byte of the key. Continue in this fashionuntil the text is exhausted. If the key is shorter than the text, itshould be recycled (start over from the beginning).For instance, xorring the text `everything remains raw` with the key`word up` should give you the following hexadecimal ciphertext:`121917165901181e01154452101d16061c1700071100`.What is the ciphertext of the hex-encoded text `the world is yours` against the key `illmatic`?<jupyter_code>from itertools import cycle

def xor(clear_text: bytes, key: bytes) -> bytes:
    # fancy way how to do key[i % len(key)]
    key_generator = cycle(key)
    # we don't like for cycles, so let's zip bytes from clear text and key
    return bytes([x^y for x,y in zip(clear_text, key_generator)])

# test part
clear_text = txt2bin('everything remains raw')
key = txt2bin('word up')
expected_cipher_text = hex2bin('121917165901181e01154452101d16061c1700071100')
cipher_text = xor(clear_text, key)

assert cipher_text == expected_cipher_text
print(f'Generated cipher text matches expected: {cipher_text == expected_cipher_text}')

# now lets get real data
clear_text = txt2bin('the world is yours')
key = txt2bin('illmatic')
cipher_text = xor(clear_text, key)
print(f'Hex Ciphertext: {bin2hex(cipher_text)}')<jupyter_output>Generated cipher text matches expected: True
Hex Ciphertext: 1d04094d161b1b0f0d4c051e410d06161b1f<jupyter_text>Exercise 2: decrypt single-letter xorThe following hex-encoded ciphertext was encoded against thesingle-letter key `$`, i.e. ASCII 36.    404b48484504404b48484504464d4848045d4bBefore decrypting, shortly explain what pattern(s) are present in thisciphertext due to the weak mode of encryption.Then, decrypt the text. What is the plaintext?<jupyter_code>cipher_text = hex2bin('404b48484504404b48484504464d4848045d4b')
print(cipher_text)<jupyter_output>b'@KHHE\x04@KHHE\x04FMHH\x04]K'<jupyter_text>When we print the cipher text in binary, we can see pattern `KHHE` repeating two times and `\x04` repeating three times. This suggests that there is one word two times and one symbol three times. Also, `HH` is repeating three times so there's repeating symbol as well.<jupyter_code>key = txt2bin('$')

print(f'Clear text: "{bin2txt(xor(cipher_text, key))}"')<jupyter_output>Clear text: "dolla dolla bill yo"<jupyter_text>And yes, now we can map that `KHHE` was `dolla`, repeating twice, `\x04` was space and `HH` was `ll`.  Exercise 3: hand crack single-letter xorThe file `text1.hex` contains a hex-encoded ciphertext that wasxor encoded with a single letter.Decrypt it. What is the first line? *(I think it is just easier to go right into excercise **4**)*  Exercise 4: automate cracking single-letter xorSolve the previous exercise, but instead of searching for the correctkey/plaintext with your eyes, make the computer do it. In other words,you should have a function that, given a single-letter xor encodedciphertext, will return you the single-byte key (and, if you want, theplaintext).You could devise a scoring function that checks, for a givendecryption, if it seems like English. Then just iterate through allpossible keys and return the key whose decryption gets the bestscore.<jupyter_code># https://inventwithpython.com/hacking/chapter20.html
FREQUENCY_OF_ENGLISH_LETTERS = {'E': 12.70, 'T': 9.06, 'A': 8.17, 'O': 7.51, 'I': 6.97, 'N': 6.75, 'S': 6.33, 'H': 6.09, 'R': 5.99, 'D': 4.25, 'L': 4.03, 'C': 2.78, 'U': 2.76, 'M': 2.41, 'W': 2.36, 'F': 2.23, 'G': 2.02, 'Y': 1.97, 'P': 1.93, 'B': 1.29, 'V': 0.98, 'K': 0.77, 'J': 0.15, 'X': 0.15, 'Q': 0.10, 'Z': 0.07}
import string
from collections import Counter

def contains_only_printable_chars(clear_text: str) -> bool:
    # check if there're any non printable characters
    return all([char in string.printable for char in clear_text])

def english_frequency_analysis(clear_text: str, diff = 5) -> float:
    if not contains_only_printable_chars(clear_text):
        return float("inf")

    # now we need only text that can be used for frequency analysis
    normalized = [l for l in clear_text.upper() if l in FREQUENCY_OF_ENGLISH_LETTERS.keys()]
    
    counter = Counter(normalized)
    
    # calculate relative frequency of letters
    relative_frequencies = {letter: (100*count)/len(normalized) for letter, count in counter.items()}
    # and calculate final score for the text
    return sum([abs(FREQUENCY_OF_ENGLISH_LETTERS[letter] - freq) for letter, freq in relative_frequencies.items()])

# returns key, score, plaintext
def decrypt_cipher(cipher_text: bytes, alphabet: [bytes]) -> [str, float, str]:
    score = {}
    plain_texts = {}
    for key in alphabet:
        plain_texts[key] = bin2txt(xor(cipher_text, key))
        score[key] = english_frequency_analysis(plain_texts[key])
        
    key = min(score, key=score.get)
    return bin2txt(key), score[key], plain_texts[key]
with open("text1.hex") as f:
    cipher_text = hex2bin(f.read())

# generate all possible keys
alphabet = [txt2bin(letter) for letter in string.ascii_letters]

key, score, plain_text = decrypt_cipher(cipher_text, alphabet)

print(f'Encryption key: "{key}" with frequency score {score}')
plain_text.split("\n")[0]<jupyter_output>Encryption key: "M" with frequency score 26.15073394495413<jupyter_text>Exercise 5: crack multiple-letter xor with given key lengthThe file `text2.hex` contains a hex-encoded ciphertext that was xorencoded against a multiple-letter key -- just like Ex 1.Crack it. You are given the following indication: the key contain10 characters.Notice that by a simple manipulation of the ciphertext, the 10-letterencryption is nothing more than a collection of 10 single-letterencryptions -- which you can already crack thanks to Ex 4.What is the key, what is the first line?<jupyter_code>def decrypt_cipher_chunking(cipher_text: bytes, key_length: int, alphabet: [bytes]):
    cipher_text_chunks = [cipher_text[i::key_length] for i in range(key_length)]
    keys = []
    for chunk in cipher_text_chunks:
        key, score, plain_text = decrypt_cipher(chunk, alphabet)
        keys.append(key)
    
    key = ''.join(keys)
    return key, bin2txt(xor(cipher_text, txt2bin(key)))
with open("text2.hex") as f:
    cipher_text = hex2bin(f.read())

alphabet = [txt2bin(l) for l in string.printable]
key, plain_text = decrypt_cipher_chunking(cipher_text, 10, alphabet)
print(f'Key: {key}')
plain_text.split('\n')[0]<jupyter_output>Key: SupremeNTM<jupyter_text>Exercise 6: crack multiple-letter xor with unknown key lengthDecrypt `text3.hex`, except this time you don't know the keylength.Even better if you can make your code find out the keylength beforetrying to decrypt.What is the key, what is the first line?<jupyter_code>with open("text3.hex") as f:
    cipher_text = hex2bin(f.read())

score = {}
plain_texts = {}
keys = []

for length in range(2, 258):
    print(f'Testing key length {length}...')
    key, plain_text = decrypt_cipher_chunking(cipher_text, length, alphabet)
    score[key] = english_frequency_analysis(plain_text)
    plain_texts[key] = plain_text
    keys.append(key)
    # trashold check, it is unlikely that english text would
    # have score higher then this, and even if it has
    # we select by min anyway at the end
    if score[key] < 50:
        break
    
key = min(score, key=score.get)
print(f'Key: {key}')
plain_text.split('\n')[0]<jupyter_output>Testing key length 2...
Testing key length 3...
Testing key length 4...
Testing key length 5...
Testing key length 6...
Testing key length 7...
Testing key length 8...
Testing key length 9...
Testing key length 10...
Testing key length 11...
Testing key length 12...
Testing key length 13...
Testing key length 14...
Testing key length 15...
Testing key length 16...
Key: CL4SS!C_TIM3L35S
{'fraction_non_alphanumeric': 0.10430715684141832, 'fraction_numerical': 0.04209266913204264, 'mean_word_length': 4.728971962616822, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '20414', 'license': 'MIT', 'n_tokens_mistral': 3270, 'n_tokens_neox': 2959, 'n_words': 1060, 'path': '02/solution_02.ipynb', 'repo_name': 'LukasForst/KBE', 'size': '9203'}