<jupyter_start><jupyter_text><NAME>  Code DemoThis notebook only calls functions that run the actual routines. This takes approximately 1.5 hours running in parallel with 8 cores to train. The evaluation can take up to 5-10 minutes if there are many clusters and the allocation is done with a neural net. We have always run our code on Windows machines. A Requirements.txt in the main directory defines the python environment used to run all our code. Link to GIT repo: https://github.com/aco8ogren/Tentin-QuarantinoNOTE: Unfortunately, the "verbosity" feature does not work in Jupyter Notebooks since the main kernal will not collect text printed by the worker cores. As such, this jupyter notebook won't give updates on the progress of the SEIIRQD training that is occuring in parallel. This is a purely cosmetic issue; you just won't know how far along the code is but everything is still running correctly. Results will still be printed in the cell as expected. The verbosity does work when run as a regular python script or in ipython.  Import relevant functions<jupyter_code>import os
import sys

HomeDIR='Tentin-Quarantino'
wd=os.getcwd()
DIR=wd[:wd.find(HomeDIR)+len(HomeDIR)]
os.chdir(DIR)

# %% Imports and path setup

# NOTE: Apparently we still need to shield. Can't say I understand why...

sys.path.append(os.getcwd())

from Dan.EpidModel_parallelized_Counties import SEIIRQD_model
from Alex.copy_of_evaluator import evaluate_predictions
from Dan.format_sub import format_file_for_evaluation<jupyter_output><empty_output><jupyter_text>Define training parameters<jupyter_code>#-- Flag to choose whether to train the model
    # If this is true, the output file from this run will be used for
    # the remainder of the sections
isTrainModel = True

#-- Define control parameters
# Flag to choose whether to save the results or not
isSaveRes = True
# Filename for saved .npy and .mat files (can include path)
    # Make sure the directory structure is present before calling
    # NOTE: when clustering, the .mat filename will be used for saving the cluster file
sv_flnm_mat = r'Sample_Model_Runs\Run.mat'
sv_flnm_np  = os.path.splitext(sv_flnm_mat)[0] + '.npy'


#-- Multiprocessing settings
# Flag to choose whether multiprocessing should be used
isMultiProc = True
# Number of cores to use (logical cores, not physical cores)
workers = 10


#-- Filtering parameters
# Threshold of deaths at and below which a COUNTY will not be trained on
    # Filters which COUNTIES are looped over in optimization/minimization loop
D_THRES = 97
# Last day used for training (good for testing)
    # must be a valid pandas.to_datetime() string
    # OR: leave as None to train until the latest data for which there is data
train_til = '2020-05-22'
# Minimum deaths considered in training
    # Sets the first DAY which will be calculated as part of the optimization
    # by only including days with more than this many deaths. THIS IS DIFFERENT than 
    # D_THRES. D_THRES selects which counties are trained on and train_Dfrom selects 
    # which DAYS are used for the optimization
train_Dfrom = 7
# Minimum number of days required for a county to be trained on
    # After filtering using train_Dfrom and D_THRES, this makes sure that there
    # are at least min_train_days worth of days to train the model on (for fit_leastsqz)
min_train_days = 5


#-- Clustering settings
# Enable clustering: combines the low-death counties into clusters for training
    # When False, the code will run as it used to
isCluster = True

cluster_max_radius = 0


#-- Sub-select counties to train on
# Flag to choose whether to sub-select
isSubSelect = False
# List of counties which should be considered
    # NOTE: This just removes ALL other counties from the df as soon as it can
just_train_these_fips = [36061, 6037]
    

#-- Method used for choosing initial conditions
    # True: Use the same vector (hardcoded) as the initial conditions for all counties
    # False: Calculate unique initial conditions for each county 
isConstInitCond = True
# When calculating unique conditions for each county, define fudge factors:
init_vec = (4.901,          # T : Is = T*cases      Old: 3.933
            0.020,          # R : Ia = R*Itot       Old: 0.862
            0.114)          # F : E  = F*Is         Old: 3.014


#-- When not multiprocessing, enable bokeh plotting (since won't cause issue)
# Flag to stating whether to plot. This only matters when not multiprocessing (isMultiProc=False)
    # When isMultiProc=True, bokeh will cause errors so we ignore this flag
isPlotBokeh     = False


#-- Set verbosity for printing
#-- Verbosity explanation:
# There are multiple levels of verbosity based on the provided integer
#   0 :     No print statements are executed
#   1 :     Only total time is printed
#   2 :     Only prints in main function are shown (those in par_fun are suppressed)
#   3 :     (DEFAULT) All print statements are executed
# *** Error-related prints are always printed
verbosity = 3

#-- Set hyperparameters
p_err_frac = 0.097   # The size of the uncertainty that we have on our optimal SEIIRQD parameters. This affects the size of our quantile differences.
death_weight = 5   # The weight with which we multiply the death error in SEIIRQD optimization. The death data is trusted death_weight times more than the symptomatic infected data.
alpha = 0.00341564933361549         # alpha of the LeakyReLU for modifying the symptomatic infected error. i.e. if alpha = 0 ==> no penalty for overestimating Sympt Inf. alpha = 1 ==> as much penalty for overestimating as underestimating.<jupyter_output><empty_output><jupyter_text>Define Formattin Parameters<jupyter_code>#-- Flag to choose whether to format a model's .mat output file
isFormat = True

#-- Define control parameters
# Day to use for allocating county deaths
alloc_day=train_til
# Flag to translate cummulative data to daily counts
isComputeDaily = True


#-- Options for allocation using Naive method
# Flag to distribue cluster deaths amongst counties naively
isAllocCounties = False
# Allocating using the mean number of num_alloc_days days BEFORE alloc_day
num_alloc_days=5

#-- Options for allocation using Neural Net
# Flag to distribute deaths with neural net
isAllocNN=True
# Number of days of death inputs
numDeaths=2
# Number of days of cases inputs
numCases=2
# Number of days of mobility inputs
numMobility=2
# Number of days of deaths outputs to average over
lenOutput=5
# Flag to remove data points with zero deaths for inputs and output
remove_sparse=True
# Number of epochs with no decrease in validation loss before training stops
Patience=4
# Dropout rate for dropout layers between the two hidden dense layers
DropoutRate=0.15
# Flag to retrain neural net or just look for model in directory
retrain=True
# Directory to save or load model from
modelDir=r'Sample_Model_Runs\NN_Model'


#-- When a model was not trained, provide filename to format
    # if a model was trained, that filename will automatically be used
format_flnm_in = r'Sample_Model_Runs\Run.mat'

#-- Provide filename for output file 
format_flnm_out = os.path.splitext(format_flnm_in)[0] + '.csv'<jupyter_output><empty_output><jupyter_text>Define Evaluation Parameters<jupyter_code>#-- Flag to choose whether to evaluate a .csv file
isEval = True


#-- When model was not formatted, provide a filename to evaluate
    # if a model was formatted, that filename will automatically be used
eval_flnm_in = r'Sample_Model_Runs\Run.csv'

#-- Day from which we should evaluate 
    # in format 'YYYY-MM-DD'
eval_start_day = '2020-05-23'

#-- Day until which we should evaluate
    # in format 'YYYY-MM-DD'
    # Set to None to evaluate until most recent day of data
eval_end_day = None<jupyter_output><empty_output><jupyter_text>Set correct file paths<jupyter_code>#-- If the user says to train a model but doesn't save the result, we can't
    # run the remaining sections since we won't have the results to format/eval
if isTrainModel and (not isSaveRes):
    if isFormat:
        raise ValueError("isTrainModel=True but isSaveRes=false so we can't format the file.")
    if isEval:
        raise ValueError("isTrainModel=True but isSaveRes=false so we can't evaluate the file.")

#-- If the user trains a model, use the output to format
if isFormat and isTrainModel:
    format_flnm_in = sv_flnm_mat
    format_flnm_out = os.path.splitext(format_flnm_in)[0] + '.csv'

# If the user formats a model, use the output to evaluate
if isEval and isFormat:
    eval_flnm_in = format_flnm_out<jupyter_output><empty_output><jupyter_text>Define full run function<jupyter_code># Needed as a function for the optional hyperparameter optimization 
if __name__ == '__main__':
    def runFull(init_vec):
        if isTrainModel:
            print('\n\n------ Training Model ------')
            SEIIRQD_model(HYPERPARAMS = (p_err_frac,D_THRES,death_weight,alpha),
                            isSaveRes = isSaveRes,
                            sv_flnm_np=sv_flnm_np,
                            sv_flnm_mat = sv_flnm_mat,
                            isMultiProc = isMultiProc,
                            workers = workers,
                            train_til = train_til,
                            train_Dfrom = train_Dfrom,
                            min_train_days = min_train_days,
                            isSubSelect = isSubSelect,
                            just_train_these_fips = just_train_these_fips,
                            isPlotBokeh = isPlotBokeh, 
                            isConstInitCond = isConstInitCond,
                            init_vec = init_vec,
                            verbosity = verbosity,
                            isCluster = isCluster, cluster_max_radius = cluster_max_radius)
            if isSaveRes:
                print('*** Model results saved to:\n    %s\n    %s'%(sv_flnm_mat, sv_flnm_np))
                


        if isFormat:
            print('\n------ Formatting File ------')
            print('*** Input filename:\n    %s'%format_flnm_in)

            format_file_for_evaluation( format_flnm_in,
                                        format_flnm_out,
                                        isAllocCounties = isAllocCounties,
                                        isComputeDaily = isComputeDaily,
                                        alloc_day=alloc_day,
                                        num_alloc_days=num_alloc_days,
                                        isAllocNN=isAllocNN,
                                        retrain=retrain,
                                        numDeaths=numDeaths,
                                        numCases=numCases,
                                        numMobility=numMobility,
                                        lenOutput=lenOutput,
                                        remove_sparse=remove_sparse,
                                        Patience=Patience,
                                        DropoutRate=DropoutRate,
                                        modelDir=modelDir)

            
            print('*** Formatted file:\n    %s'%format_flnm_out)
            

        if isEval:
            print('\n------ Evaluating File ------')

            
            print('*** Input filename:\n    %s'%eval_flnm_in)
            print('\n\n')
            
            score = evaluate_predictions(eval_flnm_in,
                                            eval_start_day,
                                            end_date = eval_end_day)
            return score<jupyter_output><empty_output><jupyter_text>Run runFull funtion to train, format, and/or evaluate the model.<jupyter_code>if __name__ == '__main__':
    runFull(init_vec)<jupyter_output>g FIPS 48119.0
Processing FIPS 48121.0
Processing FIPS 48123.0
Processing FIPS 48125.0
Processing FIPS 48127.0
Processing FIPS 48129.0
Processing FIPS 48131.0
Processing FIPS 48133.0
Processing FIPS 48135.0
Processing FIPS 48139.0
Processing FIPS 48141.0
Processing FIPS 48143.0
Processing FIPS 48145.0
Processing FIPS 48147.0
Processing FIPS 48149.0
Processing FIPS 48151.0
Processing FIPS 48153.0
Processing FIPS 48157.0
Processing FIPS 48159.0
Processing FIPS 48161.0
Processing FIPS 48163.0
Processing FIPS 48165.0
Processing FIPS 48167.0
Processing FIPS 48169.0
Processing FIPS 48171.0
Processing FIPS 48173.0
Processing FIPS 48175.0
Processing FIPS 48177.0
Processing FIPS 48179.0
Processing FIPS 48181.0
Processing FIPS 48183.0
Processing FIPS 48185.0
Processing FIPS 48187.0
Processing FIPS 48189.0
Processing FIPS 48191.0
Processing FIPS 48193.0
Processing FIPS 48195.0
Processing FIPS 48199.0
Processing FIPS 48203.0
Processing FIPS 48205.0
Processing FIPS 48207.0
Processing FIPS 48209.0
P[...]
{'fraction_non_alphanumeric': 0.05981631211241071, 'fraction_numerical': 0.028024177721956197, 'mean_word_length': 2.692753623188406, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '249680', 'license': 'MIT', 'n_tokens_mistral': 3806, 'n_tokens_neox': 3456, 'n_words': 1374, 'path': 'Tentin_Quarantino_43_code_demo.ipynb', 'repo_name': 'aco8ogren/Tentin-Quarantino', 'size': '12750'}