<jupyter_start><jupyter_text>Findings Thus Far-----------------* Observation: a so-called "generator" has *no length*.* Quesiton: *why?* (looking at you, @amwbartosch)<jupyter_code>eager_results = list(lazy_results)
print(len(eager_results))
print(eager_results)<jupyter_output>[['0.000', '0.000', True], ['0.625', '0.000', True], ['1.249', '0.011', True], ['1.875', '0.037', True], ['3.000', '0.048', True], ['3.625', '0.062', True], ['4.249', '0.060', False], ['4.875', '0.061', False], ['5.500', '0.099', True], ['6.369', '0.116', True]]<jupyter_text>Using the clean_data "wrapper"----------------------------<jupyter_code>data = open("sample_data.tsv", "rb").read()
print(clean_data(data))<jupyter_output>b'0.000,0.000,True\r\n0.625,0.000,True\r\n1.249,0.011,True\r\n1.875,0.037,True\r\n3.000,0.048,True\r\n3.625,0.062,True\r\n4.249,0.060,False\r\n4.875,0.061,False\r\n5.500,0.099,True\r\n6.369,0.116,True\r\n'
{'fraction_non_alphanumeric': 0.30196936542669583, 'fraction_numerical': 0.175054704595186, 'mean_word_length': 4.545454545454546, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '579364', 'license': 'Apache-2.0', 'n_tokens_mistral': 497, 'n_tokens_neox': 409, 'n_words': 54, 'path': 'data_cleaning_tools_notebook.ipynb', 'repo_name': 'giphahne/bubble-tracker-data', 'size': '914'}