<jupyter_start><jupyter_text>Preparation and processing of textual dataThis notebook will recap key Python concepts for data preparation and cleaning Project 1: creating Python dictionaryIn this project we will create a frequency dictionary of N-grams and Skip-grams Tasks of the projectRead a corpus in text format and create the following frequency dictionaries, sorted in reverse order of frequencies:1. The dictionary of single words2. The dictionary of continuous word sequences (N-grams) within a sentence, from length 1 and up to N words long (N to be given as a parameter before running the programme)3. The dictionary of N-grams with N > 2, and up to N-words long, which have 1 gap in the middle (skip-grams), where the omited word is replaced by a placeholder "___". Stages of the projectStage 1: downloading and preparing the datasetStage 2: reading the inputStage 3: extracting required featuresStage 4: storing and frequency countsStage 5: generating output file, interpreting the resultsTask: create code cells for each stage, which will perform the tasks.<jupyter_code># downloading the corpus<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.040421792618629174, 'fraction_numerical': 0.01054481546572935, 'mean_word_length': 5.471590909090909, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '474721', 'license': 'Apache-2.0', 'n_tokens_mistral': 282, 'n_tokens_neox': 255, 'n_words': 163, 'path': 'session101_introduction.ipynb', 'repo_name': 'bogdanbabych/compling2021', 'size': '1138'}