<jupyter_start><jupyter_text>Info about [files to delete](https://github.com/enram/data-repository/issues/46):<jupyter_code>countries = ["be", "ch", "cz", "de", "dk", "ee", "es", "fr", "hr", "no", "pl", "si", "sk"]
start_date = datetime.date(2016, 11, 1)
end_date = datetime.date(2017, 11, 21)<jupyter_output><empty_output><jupyter_text>Loop h5 files and remove those according to criteria:<jupyter_code>files_to_delete = []
for j, file in enumerate(s3handle.bucket.objects.all()):
    if file.key.endswith(".h5"):
        file_info = parse_filename(os.path.basename(file.key))
        date = datetime.date(int(file_info["year"]), 
                             int(file_info["month"]), 
                             int(file_info["day"]))
        country = file_info["country"]
        if (country in countries) and (date > start_date) and (date <= end_date):
            files_to_delete.append(file)
            file.delete()
deleted_files = [file_info.key for file_info in files_to_delete]
deleted_files<jupyter_output><empty_output><jupyter_text>Remove corresponding .zip files:* delete set of zip-files* update the november 2017!!! TODO<jupyter_code>countries = ["be", "ch", "cz", "de", "dk", "ee", "es", "fr", "hr", "no", "pl", "si", "sk"]
year_months = ["201611", "201612", "201701", "201702", "201703", "201704", 
               "201705", "201706", "201707", "201708", "201709", "201710",
               "201711"]
for j, file in enumerate(s3handle.bucket.objects.all()):
    if file.key.endswith(".zip"):
        current_file = os.path.basename(file.key)
        country = current_file[:2]
        year_month = current_file[5:11]
        if (country in countries) and (year_month in year_months):
            print(current_file, " : " ,country, year_month)
            file.delete()
s3handle.create_zip_version(s3handle.count_enram_coverage(level="month"))<jupyter_output><empty_output><jupyter_text>Update the coverage file<jupyter_code># Rerun file list overview to extract the current coverage
coverage_count = s3handle.count_enram_coverage(level='day')
with open("coverage.csv", 'w') as outfile:
    dm.coverage_to_csv(outfile, coverage_count)
s3handle.upload_file("coverage.csv", "coverage.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1591417076441663, 'fraction_numerical': 0.04962002682163612, 'mean_word_length': 4.15668202764977, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '558944', 'license': 'MIT', 'n_tokens_mistral': 839, 'n_tokens_neox': 762, 'n_words': 156, 'path': 'file_transfer/ad_hoc_management/remove_on_date_country_selection.ipynb', 'repo_name': 'enram/data-repository', 'size': '2237'}