<jupyter_start><jupyter_text>Test Burrows Wheeler Transform (BWT) This is a simple notebook to test the implementation of the BWT.<jupyter_code>from Burrows_Wheeler_Transform import transform
from Burrows_Wheeler_Transform import inverse
import time<jupyter_output><empty_output><jupyter_text>Corectness of the implementation  First we test the correctness of the implementation for the transform using some known BWT. We can have as input a string with or without '$'<jupyter_code>finalString = transform('Banana')
if finalString == 'a$nnBaa':
    print('The BWT of the string Banana is: %s' %(finalString))


finalString = transform('mississippi$')

if finalString == 'ipssm$pissii':
    print('The BWT of the string mississippi is: %s' %(finalString))<jupyter_output>The BWT of the string Banana is: a$nnBaa
The BWT of the string mississippi is: ipssm$pissii<jupyter_text>Then we test  the correctness of the implementation for the inverse of BWT<jupyter_code>originalString = inverse('a$nnBaa')
if originalString == 'Banana':
    print('The original string is: %s' %(originalString))
    

originalString = inverse('ipssm$pissii')
if originalString == 'mississippi':
    print('The original string is: %s' %(originalString))<jupyter_output>The original string is: Banana
The original string is: mississippi<jupyter_text>Common errorThis is a list of common error that an user could make<jupyter_code># transform with more then one $
finalString = transform('mississippi$$')

if finalString == 'ipssm$pissii':
    print('The BWT of the string mississippi is: %s' %(finalString))
# inverse string without the $ character 
originalString = inverse('annBaa')
if originalString == 'Banana':
    print('The original string is: %s' %(originalString))<jupyter_output><empty_output><jupyter_text>Performance test  Now that we have test the correctness of the implementation we will test the speed of this implementation, in particular we are interesting in smoothly handle string with 1000 characters.<jupyter_code>with open('TestSample/oneThousandChar.txt') as file:
    oneChar = file.read().replace('\n', '')

 
  
with open('TestSample/twoThousandChar.txt') as file:
    twoChar = file.read().replace('\n', '')


with open('TestSample/twelveThousandChar.txt') as file:
    twelveChar = file.read().replace('\n', '')
    
with open('TestSample/twentyFourChar.txt') as file:
    twentyFourChar = file.read().replace('\n', '')
# compute the time for the transform of one thousand characters 
print('The time for the transform is: ')
%time finalString = transform(oneChar)

# compute the time for the inverse of one thousand characters 
print('The time for the inverse is: ')
%time inverseTransf = inverse(finalString)
# compute the time for the transform of two thousand characters 
print('The time for the transform is: ')
%time finalString = transform(twoChar)

# compute the time for the inverse of two thousand characters 
print('The time for the inverse is: ')
inverseTransf = inverse(finalString)
# compute the time for the transform of twelve thousand characters
print('The time for the transform is: ')
%time finalString = transform(twelveChar)

# compute the time for the inverse of twelve thousand characters 
print('The time for the inverse is: ')
%time inverseTransf = inverse(finalString)
# compute the time for the transform of twenty-four thousand characters 
print('The time for the transform is: ')
%time finalString = transform(twentyFourChar)


# compute the time for the inverse of twenty-four thousand characters 
print('The time for the inverse is: ')
%time inverseTransf = inverse(finalString)<jupyter_output>The time for the transform is: 
CPU times: user 1.21 s, sys: 451 ms, total: 1.66 s
Wall time: 1.66 s
The time for the inverse is: 
CPU times: user 1.6 s, sys: 7.79 ms, total: 1.61 s
Wall time: 1.62 s
{'fraction_non_alphanumeric': 0.09380715965508231, 'fraction_numerical': 0.007055134570159394, 'mean_word_length': 5.466216216216216, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '657210', 'license': 'MIT', 'n_tokens_mistral': 1144, 'n_tokens_neox': 1099, 'n_words': 432, 'path': 'Test BWT.ipynb', 'repo_name': 'EmanuelSoda/ProgrammingPython', 'size': '3827'}