<jupyter_start><jupyter_text>Main script This notebook coordinates the running of the main script, by calling the function 'region_loss_estimation' from main_functions. Before you can run this notebook, you must have finished the preprocessing step. You can run the main script in parallel (using multiprocessing) or for single regions<jupyter_code># -*- coding: utf-8 -*-
"""
This coordinates the main script 'region_loss_estimation'

@author: <NAME> and <NAME>
"""

import sys
sys.path.append("..") #import main project folder which is one level higher

import os as os
from pathos.multiprocessing import Pool 
import pdb
import time as time
import warnings


from main_functions import region_loss_estimation
from postproc_functions import NUTS_up, NUTS_down, NUTS_0_list
from utils_functions import load_config

warnings.simplefilter(action='ignore', category=FutureWarning)

osm_data = load_config()['paths']['osm_data']
output_path = load_config()['paths']['output']<jupyter_output><empty_output><jupyter_text>Choose the regions for which to run the script<jupyter_code>### PREPARE THE REGIONS FOR WHICH TO RUN THE MAIN SCRIPT
#regions = [(x.split('.')[0]) for x in os.listdir(osm_data)] #takes the names of the .osm.pbf files as the list of regions to compute over 

#PREFERRED OPTION 0: (preferred version in the EU version of the model for COACCH)
N0_domain = NUTS_0_list(EU28 = True, EFTA=True, CAND=True)
N0_exclude = ['MT','CY','TR','IS']
N0_domain = [elem for elem in N0_domain if elem not in N0_exclude] #35 countries remaining
regions = []
for N0 in N0_domain:
    regions.extend(NUTS_up(N0,True)) #1519 NUTS-3 regions

#OPTION 2: RUN FOR ONE COUNTRY
#super_region = 'BE'
#regions = NUTS_up(super_region,True)

#OPTION 3: RUN FOR MULTIPLE COUNTRIES
#super_regions = ['LI','NO']
#regions = []
#for super_region in super_regions:
#    regions.extend(NUTS_up(super_region,True))<jupyter_output><empty_output><jupyter_text>Run the model in parallel<jupyter_code>start = time.time()
    
### Parallel computation
if __name__ == '__main__':
    with Pool(7) as pool: #set the number of pools suitable for your pc
        pool.map(region_loss_estimation,regions,chunksize=1) 
    
end = time.time()
print("The script runned for {} seconds.".format(end-start))<jupyter_output>The script runned for 162126.0655310154 seconds.<jupyter_text>Run model for one single region<jupyter_code>### Single computation:
region_loss_estimation('PL518') 
### LINEAR COMPUTATION (ONLY USE ONE CORE)
#for region in regions: 
#    region_loss_estimation(region,log_file="region_loss_estimation_log_{}.txt".format(os.getenv('COMPUTERNAME')))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10417450169236556, 'fraction_numerical': 0.01692365550959007, 'mean_word_length': 6.150537634408602, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '94711', 'license': 'MIT', 'n_tokens_mistral': 918, 'n_tokens_neox': 873, 'n_words': 267, 'path': 'run_core_model/2_Main_multi.ipynb', 'repo_name': 'keesvanginkel/OSdaMage', 'size': '2671'}