<jupyter_start><jupyter_text>As the dataset is already preprocessed there is a little need in extended exploration. Thus my goal is to make simple check of correspondence between texts and their summaries and distribution of words/sentences quantity for samples.This analysis will be made using custom function `explore_text_and_summaries`.<jupyter_code># train data
print('EXPLORING TRAIN DATA')
explore_text_and_summaries(text_path='/prep_data/train_data_prep',
                           summaries_path='/prep_data/train_labels_prep',
                           text_word_tok=True,
                           text_word_tok_path='/temp_data/train_tokenized_word_level')
# validation data
print('EXPLORING VALIDATION DATA')
explore_text_and_summaries(text_path='/prep_data/val_data_prep',
                           summaries_path='/prep_data/val_labels_prep')
# test data
print('EXPLORING TEST DATA')
explore_text_and_summaries(text_path='/prep_data/test_data_prep',
                           summaries_path='/prep_data/test_labels_prep')<jupyter_output>EXPLORING TEST DATA
----------
Basic parameters of texts and summaries:
----------
Length of text is 11490
----------
Comparison of text to its summary:
##########TEXT##########
greece warned it will go bankrupt next week after failing to stump up enough cash to pay millions of public sector workers and its international debts . deputy finance minister dimitras mardas set alarm bells ringing yesterday when he declared the country had been ' running on empty ' since february . with a debt repayment deadline looming on may 1 , greece faces the deeply damaging prospect of having to snub its own employees to make a $ 200m payment to the international monetary fund . in a sign of the bleak climate engulfing the country , greece 's biggest tax office had its electricity cut off yesterday after the finance ministry failed to pay its power bills . at a summit in brussels today , prime minister alexis tsipras is expected to appeal to german chancellor angela merkel for more aid to avoid going b[...]
{'fraction_non_alphanumeric': 0.06983511154219205, 'fraction_numerical': 0.004364694471387003, 'mean_word_length': 3.808857808857809, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '642986', 'license': 'MIT', 'n_tokens_mistral': 565, 'n_tokens_neox': 531, 'n_words': 224, 'path': 'exploratory_notebook/basic_text_exploration.ipynb', 'repo_name': 'IevgenP/text_summarization', 'size': '2062'}