In Ontario, local public health units (PHUs) are responsible for leading case investigations, contact tracing, and follow-up. The workforce capacity and operational requirements needed to maintain this public health strategy during the COVID-19 pandemic were unprecedented.Public Health Ontario’s Contact Tracing Initiative (CTI) was established to provide a centralized workforce. This program was unique in leveraging existing human resources from federal and provincial government agencies and its targeted focus on initial and follow-up phone calls to high-risk close contacts of COVID-19 cases. By setting criteria for submissions to the program, standardizing scripts, and simplifying the data management process, the CTI was able to support a high volume of calls.During its 23 months of operation, the CTI was used by 33 of the 34 PHUs and supported over a million calls to high-risk close contacts. This initiative was able to meet its objectives while adapting to the changing dynamics of the pandemic and the implementation of a new COVID-19 provincial information system. Core strengths of the CTI were timeliness, volume, and efficient use of resources. The CTI was found to be useful for school exposures, providing support when public health measures were lifted, and in supporting PHU’s reallocation of resources during the vaccine roll-out.When considering future use of this model, it is important to take note of the program strengths and limitations to ensure alignment with future needs for surge capacity support. Lessons learned from this initiative could provide practice-relevant knowledge for surge capacity planning.The online version contains supplementary material available at 10.17269/s41997-023-00773-6. In the early days of the COVID-19 pandemic, case and contact management was identified as an important strategy for interrupting chains of transmission and reducing mortality associated with COVID-19 WHO, . Case inPublic Health Ontario’s (PHO) Contact Tracing Initiative (CTI) was established in Ontario to provide a centralized workforce leveraging existing human resources from federal and provincial agencies. The CTI was focused solely on contact notification; that is, providing initial and follow-up phone calls to high-risk close contacts of confirmed or probable cases of COVID-19. The CTI was designed to support a high volume of calls and be responsive to shifts in the need for surge capacity support.This paper describes how this program was designed and implemented, how it was used by public health units (PHUs), and overall lessons learned that could inform future models to increase capacity for contact follow-up support.Diseases of Public Health Significance, as defined by the Ministry of Health (MOH), including COVID-19. Case definitions for all Diseases of Public Health Significance and the direction for their public health management are established by the MOH. During the COVID-19 pandemic, the MOH routinely updated provincial requirements for case and contact management and monitored workforce capacity at the PHU level. PHO is a partner in Ontario’s public health system, delivering public health laboratory services and providing scientific and technical advice to support public health practice.In Ontario, 34 regional PHUs are responsible for leading case and contact management for their respective regions. There is a duty to report In mid-March 2020, the MOH requested that PHO develop a centralized program to support Ontario’s 34 PHUs with contact management. The MOH was meeting routinely with PHUs to assess capacity across a number of areas. It was quickly recognized that some PHUs would reach capacity limits to maintain provincial requirements for case and contact follow-up. PHO was asked to explore options for providing support based on available resources. The workforce of several provincial and federal agencies was offered to make daily contact follow-up calls. There were no existing protocols in place to define roles, responsibilities, and processes for a provincial contract tracing program.Surge capacity has been described as the ability of a facility or system to expand its operations to manage an influx of patients in response to an incident .The CTI required a comprehensive data management strategy to support information exchange between the COVID-19 workforce supported by government agencies, PHO overseeing data quality and reporting, and the 34 PHUs leading case investigations. In 2020, Ontario implemented a new provincial information system: the Public Health Case and Contact Management Solution (CCM). PHUs used CCM to enter details about COVID-19 cases and high-risk close contacts. PHUs could transfer contacts to the CTI using a custom assign button that would automatically generate follow-up tasks for callers. Following training and the completion of confidentiality and CCM user agreements, callers were given access to CCM. Customized user profiles were set up in CCM to simplify the transfer of information and limit access to information within CCM.Callers could select the appropriate call script and then advance through each page of the guided flow while speaking with the contact. Phone calls involved a symptom assessment and the delivery of information about testing and self-isolation. Selected answers directed the flows in CCM and, ultimately, what guidance was provided to contacts. Figure The transition to CCM started during the early periods of the pandemic in Ontario, with a focus initially on COVID-19 case and contact management and reporting. The CCM provided an essential infrastructure support and also presented significant challenges for public health professionals with the need to quickly learn the new system and adapt existing processes—all while maintaining case and contact management capacity. During the transition period, access to the CTI was interrupted for some PHUs that had not yet transitioned to CCM. When CCM was fully implemented, uptake of the CTI increased as this new system simplified the process of sending contacts to the program.Figure In the case of Statistics Canada, the organization had a large workforce of professional interviewers who were already trained and skilled in following scripts, data entry procedures, and client interactions. Existing processes were already in place to support recruitment and to manage a large workforce. Staff could be cross-trained across regional areas and on the policies of other jurisdictions. This allowed staff to support across jurisdictions as call demands changed and workload fluctuated.The CTI was designed to support a high volume of calls. This was facilitated by having set criteria for submissions to the program, standardizing scripts, automating flows, and simplifying data entry tasks. An important principle guiding the development and updates to the CTI was to standardize and simplify, where possible, call procedures and data entry requirements.Monitoring and reporting on the volume of submissions and the timeliness of contact notification (e.g. proportion of contacts reached within 24 and 48 h of submission to the program).Surveying callers and organizing weekly meetings with supervisors to identify opportunities to improve resources, training, and other supports.Documentation of milestones and changes to the program.From the beginning, the PHO team allocated resources to performance monitoring, quality improvement, and evaluation activities. This included:Early evaluation activities included outlining a logic model for the program which helped focus an evaluation strategy for the CTI. This strategy involved documentation, monitoring uptake, supporting team debriefs to identify opportunities for improvement, and building in opportunities for program stakeholders to reflect on challenges and lessons learned.In the first few months of the program, there were low levels of participation among the 34 PHUs and a lower-than-expected call volume relative to the number of staff trained. However, uptake started to increase shortly following the initial implementation phase with increased awareness of the program and with changes in capacity and demand at the PHU level. The transition to CCM also enabled more PHUs to start using the program.n = 33/34) accessed support from the CTI to reach high-risk contacts. Additional supports were implemented by the MOH in later periods of the pandemic to increase capacity for the full case and contact management process, including a workforce that could be directly assigned to PHUs if they were managing a high volume of cases and contacts or needed more complex forms of support.In the 23 months that the CTI operated, over one million phone calls were made to high-risk contacts in Ontario. Almost all PHUs . As a result, PHUs no longer required the same volume of contact notification calls to be made on a daily basis, resulting in a sharp decline in program use by the end of December 2021, and a pause to the program operations by late February 2022.A key component of the evaluation of the CTI was to gather evaluative feedback from PHU key informants to capture their perspective on strengths and limitations of the program. In the spring of 2022, the evaluation team reached out to all PHUs that had used the CTI. Online Resource Key informants emphasized that the CTI was able to help manage a high volume of calls to high-risk close contacts and provide timely support to encourage self-isolation. Another advantage was having training and resources come from a centralized source. It would have been resource intensive for each PHU to update materials or to provide training to callers whenever there was a change in guidance.Another strength was the standardized approach to call scripts. PHU participants shared that it was reassuring to know that all high-risk close contacts were receiving the same information through the program. PHUs also shared that they could use the resources developed by PHO to update their own internal resources quickly, if needed. During the first few months the CTI was operating, there were over 180 requests to adapt and/or use PHO resources and training materials. Last, PHU participants also emphasized the important role the CTI played in alleviating extensive workload pressures on staff. The program faced some challenges over its tenure. A common limitation noted by the PHUs was a delay in updates to call scripts each time provincial guidance was updated. The scripts were accessed by callers through the new information system (CCM). Changes to script language could only be made when the team responsible for CCM scheduled a system update. While having scripts embedded in the data management system helped reduce data and information errors and simplified call processes, it took longer to update and implement the scripts compared to having separate standalone documents. As guidance changes became more and more frequent, it became evident that the system needed to be more flexible to respond to these changes quickly. Relatedly, there was a learning curve around implementing a new provincial data entry system in the midst of a pandemic. All users of the system, PHU staff, CTI team at PHO, and the Statistics Canada callers, had to learn to navigate a new data management system and continually adapt as improvements were implemented. Another tension felt by users was the balance between designing a standardized approach for a centralized system across the province and the desire to have a program that could be tailored to meet the different needs of PHUs. For example, if a PHU had enhanced guidance for case and contact management that differed from the standardized script, they could not use the program. Some PHUs also noted how it was limiting that the program was not able to support more comprehensive clinical documentation on client interactions or provide support for more complex tasks in the contact management process. As noted earlier, the scope of support provided by the CTI needed to align with the resources available, including access to a workforce that did not have a medical background. The CTI provided PHUs with a timely surge capacity system which reached a high volume of high-risk close contacts in a short period of time. There were a number of lessons learned that would apply to situations where public health needs to scale case and contact management capacity quickly. One area of learning was around the importance of continuing to assess the fit between the scope and capacity of a centralized workforce and the types of supports needed by PHUs. The CTI was most valuable when PHUs had to manage sudden increases in the volume of high-risk close contacts following changes in public health measures and when managing competing priorities, and became less valuable when there was a need to tailor guidance and offer more complex forms of support. When considering whether this model could be used in the future or by another jurisdiction, it would be important to consider that this program was only able to deliver and collect standardized information .Embedding scripts and flows in the new CCM system was an innovative feature of the CTI that aimed to reduce information errors. However, this resulted in delays getting call scripts updated following guidance changes. Sharing standalone scripts could be a more nimble option when guidance and messaging need to change quickly and often.In the early development phase, we found there was a lack of guidance on how to assess the performance and effectiveness of this type of extrinsic surge capacity program. In addition, we found that conceptual frameworks seemed more helpful for understanding surge capacity for healthcare systems as opposed to more upstream public health service delivery Below is the link to the electronic supplementary material.


The escalating threat of natural disasters to public safety worldwide underlines the crucial role of effective environmental risk management tools, such as insurance. This is particularly evident in the case of earthquakes that occurred in Oklahoma between 2011 and 2020, which were linked to wastewater injection, underscoring the need for earthquake insurance. In this regard, from a survey of 812 respondents in Oklahoma, USA, we used supervised machine learning techniques , decision tree, and random forest classifiers) to identify the factors that influence earthquake insurance uptake and to predict individuals who would acquire earthquake insurance. Our findings reveal that influential factors that affect earthquake insurance uptake include demographic factors such as older age, male gender, race, and ethnicity. These were found to significantly influence the decision to purchase earthquake insurance. Additionally, individuals residing in rental properties were less likely to purchase earthquake insurance, while longer residency in Oklahoma had a positive influence. Past experience of earthquakes was also found to positively influence the decision to purchase earthquake insurance. Both decision trees and random forests demonstrated good predictive capabilities for identifying earthquake insurance uptake. Notably, random forests exhibited higher precision and robustness, emerging as an encouraging choice for earthquake insurance modeling and other classification problems. Empirically, we highlight the importance of insurance as an environmental risk management tool and emphasize the need for awareness and education on earthquake insurance as well as the use of supervised machine learning algorithms for classification problems. However, the extent to which people see and understand the consequences of natural disasters as well as the degree to which the disaster threatens their safety are rarely considered in earthquake insurance design. Survey research has suggested that the ability to comprehend the consequences of natural hazards in any location is linked to several demographic variables of respondents  as well as a variety of local contextual factors 6.The current surge in the occurrence, intensity, and severity of natural disasters  around the world has called for the need for various insurance packages as a risk management tool to guarantee the safety of nations and their citizenry7, the primary motivation for purchasing disaster insurance among North Dakotans is their perception of risk. Furthermore, Botzen and Bergh8 conducted a contingent valuation survey in the Netherlands to elicit individual risk beliefs and flood insurance, and found that individuals are willing to pay for disaster insurance based on their perception of flood risk. In the Oklahoma context in particular, Li et al.9 found that perception on the efficacy of protective action  has a significant role in risk mitigation action for tornados and earthquakes.We conjecture that these factors have played a significant role in numerous global contexts where earthquakes or other natural disasters have occurred. According to Pynn and Ljung 10 that compared flood insurance in Germany and the Netherlands found that people are inclined to insure for natural disasters such as floods based on their risk experience, risk perception, and the charity hazard. Niyibizi et al.11 found that, on average, Oklahomans prefer lower earthquake risks and support regulations on activities linked to human-induced earthquakes in their state to reduce seismic activity. In addition, Abbas et al.12 found that in Pakistan, factors such as age, non-agricultural income, and preconceptions about the effectiveness of insurance can influence an individual’s willingness to pay for flood insurance, while Oral et al.13 discovered that in Turkey, earthquake preparedness is influenced not only by an individual's experience with earthquakes but also by cultural factors. Moreover, Greer et al.14 and Choi et al.15 argue that in the U.S., political ideology could significantly affect decision-makers’ perceptions of hazards, but its effects vary by the type of hazard.A study by Seifert et al.17. An example is Hurricane Katrina, one of the most catastrophic hurricanes in recorded history, which caused damages exceeding US$100 billion18, of which only $62 billion worth of the damaged property was insured19.Taking the foregoing expositions together, insurance uptake is important because natural disasters can cause significant economic, environmental, and financial losses, impacting not only individuals but also entire communities, regions, and countries. The costs associated with natural disasters can vary greatly depending on their scale and severity, but they can include a wide range of expenses such as damage to infrastructure, homes, and businesses; environmental degradation; loss of income; loss of life and reduced economic growth20. Wang et al.21 observed that in response to the repeated earthquakes, the Chinese government launched the Catastrophe Insurance Pilot Program in Chuxiong in 2014. This program aimed to transfer the individual risk of injury and property loss to the insurance market, thus reducing the burden on the government and cutting down on expenditures20. Indirect costs, such as disruptions to supply chains and transportation networks, lost productivity, and increased insurance premiums, can also be significant. In some cases, the economic costs of natural disasters and earthquakes can be so substantial that they can destabilize national economies, particularly in developing countries with limited resources and infrastructure to cope with such events22. Consequently, it is crucial to develop strategies and policies for managing and mitigating the immediate and long-term effects of natural disasters and earthquakes.Another example is the devastating earthquake that struck Sichuan Province in China on May 12, 2008, causing over 88,670 deaths, over 374,000 injuries, and the destruction of an estimated 5.5 million structures23 and Murphy et al.24 explain that individuals’ perception on hazard and corresponding policy significantly affect not only their risk preparedness action but also their level of compliance toward hazard control policy. Moreover, Wehde and Choi25 argue that individual’s perception and trust in hazard control policy may vary depending on the type of hazard. Therefore, without accounting for potential behavioral factors on risk preparedness decision for a given disaster , it would be difficult to design an effective policy. Against the foregoing, this study has two objectives. The first is to analyze the factors that influence the decision to purchase earthquake damage insurance. The second objective is to predict which individuals are more likely to obtain earthquake insurance in Oklahoma, given the state's increased vulnerability to earthquakes from 2010 to 202027. To realize these twin objectives, we use a variety of supervised machine learning techniques, encompassing logistic regression, ridge regression, and LASSO, to identify the influential factors of earthquake insurance uptake. In addition, decision trees and random forest algorithms are utilized for predictive modeling. The primary goal of the research is to provide insights on how to design policies aimed at encouraging more individuals and businesses to obtain earthquake damage insurance, and minimizing the risk of severe damage and financial loss in the event of an earthquake. The results of this study will provide policymakers with insights on how to design and evaluate feasible methods of implementing earthquake insurance as a compensation mechanism. The findings may also help insurance companies develop more effective marketing strategies to encourage more individuals and businesses to purchase earthquake insurance. Individuals and businesses can also use the study's findings to inform their decision-making process regarding earthquake insurance purchase, potentially resulting in lower financial losses in the event of an earthquake. According to Franco28, insurance can be a powerful ‘ex-ante’ strategy in an earthquake mitigation framework when combined with other approaches, because it can help achieve other important goals such as safe building construction and land development regulations.Moreover, despite the potentially enormous cost of the hazards, each individual has a different attitude toward risk preparedness action and hazard control policy depending on their perspective on hazards. For instance, Choi and Wehde31. While the state experienced less than two M3.0 earthquakes per year on average from 1978 to 2008, the Oklahoma Geological Survey32 reported a sharp increase in earthquakes from 41 to 903 between 2010 and 2015, all which exceed M3.0 on the Richter scale. Although scientific research has provided strong evidence linking fracking to an increase in the number of earthquakes in Oklahoma, the state's risk management and protective action options against induced earthquakes have raised concerns among all individuals and businesses. The legal system's insufficient compensation for earthquake victims, as reported by Konschnick33 and Ng'ombe and Boyer34, adds to the uncertainty. Furthermore, even though most earthquakes recorded in Oklahoma between 2011 and 2016 were relatively small, some caused significant damage to homes and businesses33. For instance, the 5.7 magnitude earthquake that occurred in Prague, Oklahoma in 2011 caused injury to two people and destroyed 14 homes and is believed to be linked to wastewater injection35. These events underscore the importance of studying the determinants of earthquake damage insurance uptake in Oklahoma to mitigate the financial and physical damages caused by future earthquakes.The state of Oklahoma is a compelling case study for this research due to the significant increase in earthquakes associated with fracking between 2010 and 202037. Our study's findings also shed light on why some individuals and businesses are hesitant to purchase earthquake insurance, despite the increased risk of earthquakes in some regions. This is important because potential earthquake risks in Oklahoma are expected to remain for some time38, and this may not be particular to Oklahoma, but many other world regions are susceptible to earthquakes.The paper makes the following contributions to existing literature. First, by identifying the factors that influence earthquake insurance uptake, this study provides empirical evidence that helps explain the decision-making process of individuals and businesses regarding earthquake insurance uptake. In doing so, the supervised machine learning algorithms used, which include logit, ridge regression, and least absolute shrinkage selection operator (LASSO), are useful in that such models, ridge regression and LASSO in particular, can identify influential factors through variable selection. When applied in this context, ridge regression and LASSO help in selecting a subset of covariates (predictor variables) that have the most influence on earthquake insurance uptake. The LASSO can act as a feature selection method, keeping only the most relevant factors influencing the decision to acquire earthquake insurance while shrinking the coefficients of less importance to zero40. When compared to traditional econometric approaches, machine learning models are more suitable for prediction problems because they can improve the accuracy and efficiency of uncovering the determinants of earthquake insurance uptake37. Using decision tree and random forest algorithms for prediction allows us to uncover potential complex relationships between the features and the target variable in addition to important variables that predict earthquake insurance uptake.Second, we contribute to the earthquake prediction policy problem by identifying attributes that predict individuals who can purchase earthquake insurance based on covariates, by using robust supervised machine learning models that include decision trees and random forests. Earthquake insurance uptake may involve non-linear relationships and complex interactions among factors. Machine learning algorithms like decision trees and random forests are particularly adept at capturing such complexities, which can be critical for understanding the nuanced influences on insurance uptake34 which established how much of the earthquake damage the oil and gas industry should be liable for, in terms of what earthquake victims endure. This study focuses on the factors associated with the decision by Oklahoma residents to purchase home/property insurance against earthquakes in the state. The data were collected from the U.S. state of Oklahoma and its counties that experienced a rise in earthquakes linked to fracking. To administer the survey, it was first approved by the Oklahoma State Institutional Review Board (IRB), and then conducted through Survey Sampling International (SSI), a reputable organization that employs numerous methods to recruit respondents. The survey was conducted with 1,153 individuals through SSI, of which 813 of them successfully completed the survey. According to Ng’ombe and Boyer34, SSI has panels of potential respondents that respond to online surveys for a given fee. A series of questions on people’s attitudes toward earthquakes in the state and their demographic, and socio-economic characteristics were asked in the survey. The variables used in this study are described in Table Our data was collected from a 2017 earthquake survey that was conducted online in Oklahoma using Qualtrics. The data is part of the research by Ng’ombe and Boyert-test statistics for the mean differences of the variables between the two groups of respondents are presented in the last column of Table Independent variables are classified into two sets. The first set of independent variables comprises socio-demographics such as age, gender, and race of the survey respondents. The second set comprises a range of variables relating to respondents’ attitudes toward earthquakes in Oklahoma. Table 41. Regarding demographic factors such as race, ethnicity, education, and income, we anticipate a heterogeneous relationship with the likelihood of having earthquake insurance. For example, Ansolabehere and Konisky42 found that minority groups are more skeptical and concerned about coal and natural gas plants located near their homes, which we believe would motivate them to purchase earthquake damage insurance. In a separate study, Boudet et al.43 observed that politically conformist respondents had at least an undergraduate degree, and older respondents were more likely to support hydraulic fracturing in the United States. Also, Ng’ombe and Boyer34 found varying levels of earthquake-related liability levels that respondents would assign to fracking firms in Oklahoma for any damage, which suggests that similar varying outcomes could be expected regarding their decision to purchase earthquake insurance.The variables included in all our models are shown in Table 10. As in Ng’ombe and Boyer34, we anticipate that renters will be less likely to have earthquake damage insurance than property owners. By political affiliation, we anticipate that Democrats (Republicans) are more (less) likely to have earthquake insurance because they have shown greater (lesser) willingness to accept fracking related benefits for regulation11. Boudet et al.43 and Davis and Fisk44 observed that Democrats are more committed to controlling wastewater injection in order to prevent potential earthquakes than Republicans.Moreover, we expect respondents who have lived in Oklahoma for a longer period to more likely own earthquake insurance because they would have more previous experience with earthquakes, especially since 2009, when the state started to experience numerous earthquakes34, we included the extent of earthquake damage respondents’ residence or property incurred in the past when assessing respondents’ attitudes toward earthquakes in Oklahoma. These include minor damage, moderate damage, major damage, and other damage as defined in Table 46. However, we expect that people whose property or house incurred any earthquake-related damage will prefer to have their property insured against earthquakes, especially since most insurers have stated that they would be able to insure against both natural and man-made earthquakes in Oklahoma47.Following Ng’ombe and BoyerOther earthquake-related factors that we expect to influence the decision to have earthquake insurance include people’s concerns about earthquake damage, the state’s obligation to regulate wastewater injection, earthquake experience, beliefs, and knowledge of the importance of oil and gas companies to Oklahoma. For example, we hypothesize that those who want wastewater injection to be abruptly stopped or who prefer to have earthquake insurance are warier of the risks of earthquakes to their property and therefore more likely to purchase earthquake insurance.To achieve its objectives, this study uses the following supervised machine learning (ML) algorithms: logit model, ridge regression, LASSO, decision trees, and random forest. We chose supervised ML over alternative methods because it served our research as a classification problem. With supervised ML, the algorithm is trained to learn the mapping between the input data (predictor variables) and the output data (dependent variable), so that it can select the influential variables of the output data and depending on the model used, also make predictions on new, previously unseen data.40. On the other hand, in unsupervised ML, the algorithm is given a dataset with no pre-existing labels or outputs . The algorithm then attempts to find patterns or structure in the data on its own, without any guidance or supervision. Unsupervised learning is used in applications where the researcher is interested in discovering hidden patterns or groupings in the data, such as clustering similar customers together for targeted marketing48.Supervised machine learning is the method of choice for applications like ours, where a specific target variable must be both explained by influential input factors and predicted. In our case, we employ supervised ML to unravel the variables that shape individuals' decisions regarding earthquake insurance acquisition while also predicting which respondents are likely to make a purchase. To clarify, supervised ML effectively categorizes variables into two crucial categories: output  and input . This categorization assumes a significant correlation and, potentially, a causal relationship between the labeled input and output variables, offering a robust framework for our analytical endeavors40, the logit classifier isi, whereby i = 1, …, N, and Let the dependent variable 39, resulting in reduced coefficients. This, in turn, shrinks the coefficients of less influential variables towards zero, improving the general performance of the model40. In this study, we applied the ridge classifier and LASSO, which impose a penalty on the logit model to avoid overfitting and identify influential variables in the decision to acquire earthquake insurance.In high-dimensional settings, where the number of predictors is large, collinearity between the predictors can lead to unstable estimates and unreliable predictions. In such cases, penalized regression models are commonly employed to improve the accuracy and interpretability of the model. Penalized regression techniques, such as ridge regression and LASSO, add a constraint to the equation to regularize the modelk is the total number of penalized coefficients. However, the ridge classifier includes all the coefficients in the final model by adding a squared magnitude of the coefficient , there would be overfitting48. Increasing The ridge classifier adds a fine-tuning parameter entclass1pt{minimaentclass1pt{minima48, the LASSO’s log-likelihood to be maximized isThe LASSO offers an alternative regularization procedure in which a number of inputs is potentially eliminated from the model, thereby bypassing limitations of a ridge classifier. Introduced by Hastie et al.The LASSO algorithm adds a penalized term to the traditional logit model which sets the coefficients of less influential variables to zero. It therefore selects only those inputs that are most relevant among the potentially many variables in the databases. Cross-validation was also used to select the 49. The use of a decision tree in this study is appealing because it provides sufficient visual information to predict which individuals are likely to insure against earthquake damage or not. The decision tree model consists of nodes and branches, where each node represents a feature, and each branch represents a possible outcome for that feature. The tree is constructed by recursively splitting the data into subsets based on the values of the input features until the subsets become homogeneous with respect to the target variable48.Decision trees are a powerful and widely used tool for solving classification problems in machine learning. A decision tree is a tree-like model that uses a set of input features to make predictions about the target variable39. While decision trees maybe easier to interpret, they can suffer from overfitting, where the model becomes too complex and captures noise in the data. To address this issue, several techniques have been developed, including pruning and ensemble methods like random forests40.Regarding training the model, the process involves searching for the best features to split the data and creating a hierarchy of nodes and branches that best separates the data into different classes. In our case, this produces a tree structure that can be used to predict earthquake damage insurance uptake for the new and previously unseen data. One of the most important benefits of decision trees is their interpretability. Unlike other machine learning algorithms that can be difficult to interpret, such as random forest (next section), decision trees are simple to understand and can provide valuable insights into the decision-making process. Decision trees can also handle non-linear relationships between the input features and the target variable, making them a valuable tool in many real-world applications50. First introduced by Breiman51, random forest classifiers are an ensemble learning method that combines the outputs of multiple decision trees to make a final prediction. During the training phase, the algorithm constructs several decision trees on different subsets of the training data. At each split in a decision tree, a random subset of the input features is selected as candidates for the split. This process helps to reduce overfitting and improve the generalization ability of the model. The final prediction is made by aggregating the predictions of all individual decision trees40. The use of random forests has become increasingly popular due to their ability to handle high-dimensional data and capture complex nonlinear relationships in the data, making them an effective tool for many real-world applications.Another ML technique we use to achieve our second objective is a random forest classifier. A random forest classifier has a wide range of applications, including image classification, natural language processing, and fraud detection39 and James et al.40, random forests are robust to overfitting, can handle large datasets with high dimensionality, can capture complex nonlinear relationships between input features and output labels, and are relatively easy to use and require minimal parameter tuning. In the present study, let x be a vector of inputs described in Table y, that reveals whether an individual has his/her property insured against earthquake damage. By so doing, the training procedure for random forests applies bootstrap aggregation or bagging to tree learners . Thus, for a training set K times) with replacement of the training data sets to fit trees to the following samples.Random forests have several advantages over other classification algorithms. According to Kassambarab = 1, …, B:Sample Train a classification tree Upon completion of the training, make predictions for unseen samples For x’40.Such a bootstrapping algorithm is considered to improve model performance because it decreases model variance without increasing the bias. Therefore, an estimate of the uncertainty of prediction is the standard deviation of the prediction from all individual regression trees 40. The OOBE measures the average prediction error for each training sample 37. In general, the testing and training errors tend to level off after a certain number of trees have been fitted. As noted by Silveira et al. 37, this is the basic bagging technique for trees. However, random forests differ from this pattern in one keyway: they use a modified learning algorithm that selects a random subset of features at each candidate split during the learning process. Silveira et al.37 contend that the selection of highly predictive features can lead to correlated results in the B trees, which can affect the accuracy of the model for earthquake damage insurance. However, using a subset of features in each split, typically the square root of the total number of features, can help to prevent correlation and improve accuracy49.The optimal number of trees, B, can be determined through two methods: cross-validation or the out-of-bag error (OOBE), as described in James et al.39. This evaluation provided insights into the extent to which these models effectively identify the most influential factors affecting earthquake insurance uptake.The performance of logistic regression, ridge regression, and LASSO models was assessed by examining the magnitude and signs of the coefficients. Ridge regression and LASSO, being regularization techniques, tend to push coefficients towards zero. Consequently, variables retaining non-zero coefficients in LASSO or having relatively larger coefficients in ridge regression are considered more influentialTN are true negatives, FN are false negatives, TP are true positives, and FP are false positives. True negatives refer to when the model identifies a data point as part of the negative class, and this identification is correct . In the same context, FN, TP, and FP happen by false negative , true positive , and false positive , respectively. These metrics collectively provided insights into the decision trees and random forest models' effectiveness in correctly identifying those likely to purchase earthquake insurance.As previously mentioned, in the case of decision trees and random forests, individuals were classified as having a high likelihood of having earthquake damage insurance if their predicted probability exceeded 0.5. Therefore, to evaluate the performance of the decision tree and random forest models for predictive modeling, various performance measures were employed, including accuracy, sensitivity, specificity, and precision which are computed as:FN and FP are unlikely to happen , the denominator of Accuracy measure will be closer to the numerator. Therefore, the measure will be closer to 1. Accuracy depicts a fraction of cases correctly classified out of the total cases. Precision, also known as positive predicted value, measures the proportion of correctly predicted positive cases out of the total number of positive cases in the dataset39. Sensitivity – or recall or the true positive rate, calculates the proportion of positive cases that are correctly classified out of the total number of real positive cases in the dataset40. Finally, Specificity measures the proportion of negative cases that are correctly identified out of the total number of negative cases in the dataset. These performance measures are typically derived from a confusion matrix, which is a 2 × 2 table that summarizes the results of a machine learning algorithm by classifying observations as TP, TN, FP, and FN.Moreover, each performance measure falls within the range of 0 to 1, with higher values indicating better model performance for each metric. For instance, if glmnet, caret, rpart, and rattle to estimate all our models56. To obtain reliable estimates, we divided our data into separate training and testing sets40. The training set comprised 70% of the total observations and was used to estimate model parameters, while the remaining 30% were used for out-of-sample estimation and prediction in the testing set. We used the glmnet package to implement ridge and LASSO classifiers and determine the optimal λ values that minimized cross-validation prediction error. Specifically, we performed cross-validation to identify the optimal values of 40. We generated plots of the cross-validation error as a function of log  classifiers to identify influential factors of earthquake insurance uptake. Additionally, we took advantage of the predictive power of decision tree and random forest classifiers to identify individuals likely to invest in earthquake insurance based on relevant covariates. Our analyses yield valuable insights for policymakers, facilitating the development of more robust insurance strategies aimed at safeguarding individuals and communities from the profound repercussions of seismic events.The recent surge in natural disasters worldwide, including the increasing occurrence, intensity, and severity of earthquakes, highlights the need for effective risk management tools, such as insurance packages, to ensure the safety of nations and their citizens59. The random forest classifier had a 100% score across accuracy, precision, sensitivity, and specificity with the decision tree coming close with regard to accuracy and specificity.The logit, ridge regression, and LASSO demonstrated their ability to provide consistent results on the influential factors that affect earthquake insurance uptake. This was evidenced by the LASSO not shrinking away any factor coefficients with coefficient signs and magnitudes being close to those from the logit and ridge classifiers. On the other hand, decision trees and random forest classifiers generated robust findings by accurately predicting which individuals are likely to acquire earthquake insurance based on the relationships within the data60 have noted that the demand and supply of insurance services related to flooding are affected by various factors, including age, economic circumstances, family status, and lifestyle. Similarly, the findings about the Hurricane Katrina disaster suggest that age is an important factor in the decision to purchase flood insurance, with older residents being more likely to insure their properties against floods compared to younger residents61. Thus, policymakers should consider these age-related factors when developing strategies to promote earthquake insurance uptake, particularly targeting younger residents who may be less likely to prioritize insurance.Our findings on influential factors of earthquake insurance uptake suggest that age is a significant factor in earthquake insurance uptake, with younger residents being less likely to insure against earthquake damage initially but becoming increasingly more likely to do so as they age. This trend may be due to older individuals becoming more aware of the need for insurance as they experience changes in lifestyle and gain a greater understanding of the potential danger posed by earthquakes. The influence of lifestyle on insurance adoption is an important consideration, as it can affect the perceived value of insurance among individuals. For instance, Sidi et al.41. However, it is possible that insurance policies are often registered in the name of the male household head, which could explain this finding. In terms of ethnicity and race, our results align with our expectations: respondents of Hispanic origin are more likely to have earthquake insurance, while Asian respondents and those of two or more races are also more likely to insure against earthquakes than White respondents. This finding is consistent with Ansolabehere and Konisky42, who found that minority groups are more skeptical and concerned about such environmental disasters. It is possible that earthquakes linked to wastewater injection motivate them to have earthquake damage insurance in case of an unexpected seismicity.Surprisingly, the results of our study indicate that male respondents are more likely to have earthquake insurance than female respondents. We had initially hypothesized that women would be more cautious of environmental risks and therefore more likely to have earthquake insurance11. Wines62 noted that Democratic state lawmaker, Cory Williams, advocated for more robust action on the quakes and criticized the state's response to the surge in earthquakes. Furthermore, the study found that respondents who have lived in Oklahoma for a longer period are more likely to have earthquake damage insurance. This result is reasonable because those who have lived in Oklahoma longer would have experienced more earthquakes, making them more likely to have their property and residence insured.Our results indicate that Democrats are less likely to own earthquake insurance than Independents, contrary to what was expected. This finding is surprising given that previous studies have shown that Democrats generally support wastewater injection regulation, which has been linked to increased seismic activity in OklahomaOur study also found that renters are less likely than homeowners to have earthquake damage insurance. This is expected since renters may be less interested in seeking earthquake insurance as they can easily move away. In terms of respondents' education levels, those with higher education levels  are more likely to have earthquake damage insurance than those with lower education levels. This could be because people with higher education levels are more aware of the benefits of insurance and the risks associated with earthquake damage.61 observation of a strong association between wealth and insurance demand, as well as a preference for insuring against high likelihood, low level consequence risks. People with relatively higher incomes may have more assets and property to protect, making them more likely to invest in earthquake insurance to recover from potential losses.Respondents with higher incomes  are more likely to have earthquake insurance than those with lower incomes. This finding is consistent with Browne et al.'s63. findings on risk preparedness for tornadoes: individuals who have experienced the devastating effects of natural disasters would be more likely to seek protection through disaster insurance. A study conducted by Ivčević et al.64 in Italy found that locals were willing to invest in mitigating climate change-related threats, which supports the idea that individuals are willing to take measures to protect themselves from natural disasters. In a similar vein, Osberghaus65 and Hong66 suggested that individuals tend to take preventive measures against flood damage when they have experienced historical damage and anticipate future damage. Furthermore, respondents who were concerned about earthquakes are more likely to insure against earthquake damage than those who were not. This finding is consistent with the observations of Pothon et al.67 that individuals are more likely to insure their homes or property against earthquakes if they believe that a devastating earthquake is imminent and are concerned about potential damage. Overall, these findings suggest that personal experiences and attitudes towards natural disasters play a crucial role in the adoption of disaster insurance.We found that respondents whose property or residence experienced moderate earthquake damage or other earthquake damage are more likely to have earthquake damage insurance compared to those whose property did not suffer any damage from earthquakes. This result is consistent with Choi et al.’s 68 on aid hazards. Browne and Hoyt68 found that individuals at risk may not purchase insurance due to their expectation of receiving aid from various sources.Moreover, we found that respondents who believe the state has a responsibility to regulate the quality of the environment and the actions of fracking companies are significantly more likely to have earthquake insurance. However, we found that financial assistance provided by the government after natural disasters may be perceived as premium-free and influence demand for natural disaster insurance, which is in line with the work of Browne and Hoyt69, the presence of oil and gas companies on a property exposes it to induced earthquakes related to wastewater injection, potentially increasing the likelihood of individuals purchasing earthquake insurance.Furthermore, we found that an increase in the number of oil and gas drilling companies at respondents' residences and respondents working in the oil and gas industry is positively associated with higher odds of having earthquake insurance. This finding suggests that individuals who have experience with oil and gas companies may have a greater awareness of the potential risks associated with fracking and may be motivated to mitigate the risk of earthquake damage by purchasing insurance. In contrast to flood insurance, where proximity to flood-prone areas may not affect an individual's decision to purchase insurance29 study in Ghana found that homeowners with internal locus control of location are more likely to insure against natural disasters. Similarly, Osberghaus65 observed in Germany that homeowners’ likelihood of insuring against flood damage increases with historical damage and the prospect of future damage. Yu et al.70 found that factors such as households' risk perception, education level, and profession significantly affect the intention to adapt to earthquake risks in rural China.Our findings on the effect of individuals’ location, experience, and risk perception on insurance uptake are consistent with previous studies. For example, Antwi-Boasiako's71 on the Georgia Coast found that risk tolerance, attitude towards community risk management initiatives, wealth exposure, and perceived damages all determine the adoption of disaster insurance. Athavale and Avila44 suggest that the demand for earthquake insurance increases with the level of risk, indicating that homeowners take earthquake insurance due to the potential risks associated with earthquakes.In addition, risk tolerance is a crucial factor that influences the uptake of disaster insurance among homeowners. Landry and TurnerWith regards to predicting which individuals are more likely to acquire earthquake insurance, most results from the decision tree are consistent with factors identified by a random forest classifier that produced perfect predictions and ranked variables by their importance in influencing earthquake insurance uptake. Our decision tree results showed that individuals who rent property and have a limited number of oil and gas companies operating on their premises demonstrate a relatively low likelihood (7%) of insuring their property. However, this subgroup constitutes a significant portion of the dataset, accounting for 50% of the observations. In contrast, individuals with the same renting status but with a higher number of oil and gas companies on their property exhibit a much higher probability (100%) of obtaining earthquake insurance, despite comprising a smaller percentage of the dataset (0%). This implies that the presence of these companies significantly influences insurance decisions, a variable that the random forest classifier also indicated to be important in predicting who acquires earthquake insurance.Moreover, individuals who do not stay on rented property, have resided in Oklahoma for less than 9.5 years, and do not receive gas leases from oil and gas companies have a 12% probability of insuring their property, with a considerable representation of 21% in the dataset. Age plays a crucial role within this subgroup. Those under 48 years old display a 7% probability, while those older than 52 years old are much less likely (0%) to insure their property. Interestingly, individuals between 48 and 52 years old in this group are highly inclined to obtain earthquake insurance (100%). Another significant factor that influences insurance decisions is the perception of wastewater injection practices. For individuals with characteristics similar to the aforementioned group but who do not believe wastewater injection should be abruptly halted and have an annual income between $75,000 and $99,999, the probability of insuring their property is 0%. In contrast, individuals with the same characteristics but who are older than 19 years old are much more likely (88%) to acquire earthquake insurance.The findings continue to underscore the complexity of insurance decision-making. Variables such as education level and duration of Oklahoma residency also come into play. Notably, individuals who have lived in Oklahoma for over 9.5 years, hold an associate degree, and do not anticipate working for oil and gas companies have a 12% probability of insuring their property, while those with similar characteristics but with job expectations in the oil and gas industry have a 100% probability.72 and Outreville73 emphasized the role of risk perceptions and external factors in insurance choices. Moreover, in the context of environmental risk perceptions, Slovic74 highlighted the impact of public attitudes and beliefs on risk-related decisions. The results from this study further underscore the significance of individual beliefs, experiences, and contextual factors in earthquake insurance uptake.These findings are consistent with prior research in behavioral economics and insurance decision-making. Studies such as Doherty and Schlesinger65,) highlight the importance of understanding individuals' risk perception, knowledge, and experience with natural disasters when examining their decision to purchase disaster insurance.Overall, the findings presented in this study could inform future policy decisions aimed at mitigating the impact of natural and induced hazards in Oklahoma and other regions facing similar risks. Collectively, our study and others 3, raising awareness about the importance of financial preparedness may include communicating about the expected allocation of disaster costs, particularly who is responsible for those costs, as well as providing information about the availability and key characteristics of disaster risk financing tools. It is therefore recommended that policymakers and stakeholders take proactive measures to increase awareness of the importance of earthquake damage insurance and work to create policies that encourage more residents to obtain insurance coverage.Based on our study's findings, we draw the following conclusions and policy implications. First, the survey results showed that only 14.4% of the total sample had earthquake damage insurance in Oklahoma, which is cause for concern given the increased likelihood of earthquakes and seismic activity in the stateSecond, the empirical findings derived from logit, ridge regression, and LASSO, focusing on the determinants of earthquake insurance adoption, are consistent with the results obtained from decision tree and random forest classifiers. However, it's worth noting that the decision tree analysis goes a step further by delving into intricate nonlinear relationships, a dimension that logit, ridge, and LASSO are inherently limited to exploring. Essentially, we found that many socio-demographic factors affect people's decisions to insure against earthquakes. Women and younger respondents were less likely to have earthquake insurance, underscoring the need for targeted awareness campaigns aimed at these groups. Additionally, respondents of Hispanic or Asian descent and those identifying as multiracial were more likely to have earthquake insurance than White respondents. it is therefore important to encourage earthquake-prone residents of all races and ethnicities to insure against earthquakes, not just targeting awareness programs to minority racial ethnic groups.76. We note that higher-income groups are more likely to insure against earthquakes due to the value of their assets.Third, our study emphasized the role of education, as respondents with higher levels of formal education were more likely to have earthquake insurance showing the potential importance of knowledge and decision analysis skills in making life choices such as earthquake insurance uptakeFourth, there are several variables related to respondents’ attitudes toward earthquakes that significantly explain people’s decision to have earthquake insurance. Respondents who have experienced moderate property damage or consider earthquakes a safety issue consider earthquake insurance and the regulation of wastewater injection as important measures that could mitigate earthquake risks and minimize mental concerns going forward. Additionally, the finding that respondents connected to the fracking industry are still interested in purchasing earthquake insurance highlights the importance of an efficient insurance system that compensates earthquake victims. Therefore, our results suggest that state regulation of activities linked to earthquakes and overseeing an effective insurance system should be a priority for policymakers to address the risks of earthquakes in Oklahoma and elsewhere.Moreover, our analysis demonstrates that both decision trees and random forests offer robust predictive capabilities for identifying individuals who would and would not acquire earthquake insurance. The decision tree achieved an accuracy rate of approximately 90% in correctly classifying these groups, while the random forest excelled with an impressive accuracy rate of 100%. Delving into precision, the decision tree classifier exhibited a commendable score of 83.58%, signifying its ability to accurately predict 83.58% of those with earthquake insurance. Intriguingly, the random forest classifier achieved a remarkable perfect precision score of 100%. While both methods are effective, the random forest's exceptional precision and robustness make it an encouraging choice for modeling earthquake insurance uptake and other classification problems, offering heightened confidence in its predictions.


The need to improve career development and training for residential aged care workers in Australia to achieve required essential competencies, including infection prevention and control competencies, has been repeatedly highlighted. In Australia long-term care settings for older adults are known as residential aged care facilities (RACFs). The COVID-19 pandemic has brought to light the lack of preparedness of the aged care sector to respond to emergencies, and the urgent need to improve the infection prevention and control training in residential aged care facilities. The government in the Australian State of Victoria allocated funds to support older Australians in RACFs, including funds toward infection prevention and control training of RACF staff. The School of Nursing and Midwifery at Monash University addressed some of these challenges in delivering an education program on effective infection prevention and control practices to the RACF workforce in Victoria, Australia. This was the largest state-funded program delivered to RACF workers to date in the State of Victoria. The aim of this paper is to provide a community case study, where we share our experience of program planning and implementation during early stages of the COVID-19 pandemic and lessons learned. In Australia long-term care settings for older adults are known as residential aged care facilities (RACFs). Globally, studies indicate that even before the coronavirus COVID-19) pandemic, RACFs were the most vulnerable institutions in terms of high incidence of infectious disease and suboptimal infection prevention and control (IPC) procedures 9 pandemi, 3. SARSIPC expertise in Australian RACFs is limited; and the COVID-19 pandemic highlighted significant gaps in IPC practices in facilities . A recenIn Australia, The Royal Commission into Aged Care Quality and Safety  special report into COVID-19 in aged care  highlighStaff training is important for effective IPC practices in RACFs . InfectiThe COVID-19 pandemic highlighted the lack of preparedness of the aged care sector in Australia to respond to emergencies and the urgent need to improve IPC training in RACFs among other challenges . The VicSince the beginning of the COVID-19 outbreak in Australia, residents in RACFs were considered at a high risk of COVID-19, leading to illness and loss of life , 5, 7. DBased on the 2020 National Aged Care Workforce Census , 70% of The Coronavirus (COVID-19) Plan for the Victorian Aged Care Sector for Victoria developed by the Department of Health and Human Services Victoria (DHHS) (The Victorian Government is committed to providing infection control training for the aged care workforce to help them adapt to the risky and changing environment posed by COVID-19. a (DHHS)  providesMonash University, the owner of the program, is Australia’s largest university; and the School of Nursing and Midwifery is ranked 5th in the Academic Ranking of World Universities by Shanghai Ranking in 2022The overall aim of this practical education program was to improve RACF staff IPC knowledge and practice, specifically related to appropriate donning and doffing of PPE to prevent transmission of COVID-19.This program incorporates three components: (1) practical face-to-face education on IPC knowledge and practices, including the application of PPE; (2) a train-the trainer model to facilitate sustainability of the program via facility champions; (3) and a virtual reality simulation, designed specifically for the aged care sector and used to consolidate knowledge.Due to the nature of the evolving coronavirus pandemic, the education program was iteratively reviewed and updated to ensure alignment with both Australian and Victorian State Government guidelines and advice.With a focus on practical application of PPE, the face-to-face component of the program addressed the following key concepts related to infection prevention: COVID-19 transmission routes, current COVID-19 pandemic concerns specific to the aged care sector, and the role of health workers in prevention of COVID-19 transmission. The use of standard infection prevention precautions, including hand hygiene, face masks and physical distancing, and their effectiveness in preventing COVID-19 transmission were also covered. Further to this, the use of transmission-based infection prevention precautions, including different levels of PPE required and situations when it is used were a focus. Finally, the correct sequencing for donning and doffing PPE to avoid contamination of self, residents, or the RACF environment were also included in this program component.This practical education session was conducted using contemporary education practices, including guided group discussion, active learning activities, and role play simulation. Pre- and post-session knowledge quizzes and observation of donning and doffing PPE using a structured checklist to assess each participant’s PPE application were used as assessments for learning. The education session was underpinned by a detailed lesson plan, which was used by all educators to promote quality and consistency in program delivery.Following the practical component of the education program, additional education was provided to key RACF staff, nominated as IPC Facility Champions. This train-the trainer model was adopted as an effective strategy to equip the appointed Facility Champions with the ability to educate others in their organization. The core advantage of a train-the-trainer model is its cost and time effectiveness when providing education to large numbers, and a greater acceptance of content delivered by internal trainers enabling the facility to have an up to date content expert to assist with day to day challenges.The train-the-trainer education session focused on the organization of the training sessions in RACFs, education practices during the training session, and accessing follow up support. Facility Champions were also able to further clarify IPC knowledge and practice if required. All education resources used during the face-to-face session, such as lesson plan, PowerPoint presentation, and handout materia were provided to the Facility Champions. Facility Champions were asked to upload a list of RACF staff they conducted training with to Monash University at the end of each session.via email for ongoing support as required; and all requests were attended within two business days.The Monash University education program coordinator contacted Facility Champions following their face-to-face session to provide support with ongoing training for their RACF staff. Facility Champions were able to contact the University training team To consolidate knowledge and build on sustainability of the education program, an online competency-based virtual reality simulation (VRS) was developed. The Monash University team worked closely with a commercial company with expertise in immersive technology and together a custom-built program of practice simulations was created. The VRS leveraged an advanced conversation engine allowing learners to have conversations with characters using artificial intelligence (AI). Mimicking real scenarios that aged care workers would face during healthcare delivery, the VRS facilitated consolidation of learning and complemented the face-to-face education. Designed purposefully and specifically for the aged care workforce, the VRS was accessible through a Windows-based personal computer PC or MAC. Enabling unscripted conversations between participants and AI characters that speak, listen, interact and are designed to replicate aged care staff, the VRS aimed to further support competency development and continued learning. On entering the VRS, participants were introduced to a simulated RACF environment and presented with a series of realistic clinical scenarios along with three different AI characters, a registered nurse, a PCA and an auxiliary staff member. The scenarios posed cases that required participants to make decisions related to prevention of infection, e.g., which level of PPE is required and the sequence of their use. Each AI character required instruction in the selection of appropriate PPE (transmission-based or standard precautions), and in the safe donning and doffing of PPE.The VRS provided RACF staff with unlimited opportunities to practice instructions in donning and doffing of PPE, in a safe, low-risk setting, and to receive real-time feedback on the accuracy of their instruction, confirming information when it was provided correctly, and correcting inaccuracies. The VRS continues to be a sustainable source of relevant information, requiring fewer human resources while ensuring the quality of training delivery and enabling a depth of understanding. Staff are able to access the platform at any time of the day or night, including weekends. The VRS has been purposefully designed to be engaging for people of all language backgrounds and literacy levels, with cases tailored to ensure they reflect the diversity of the RACFs workforce. As such, the platform provided an inclusive and sustainable risk management strategy.via potential rapid delivery of new modules during times of crisis.This education paradigm was chosen by the project team because it is engaging for users, consolidates learning, and assesses decision-making. This paradigm also has the added advantage of creating the basis for potential transformation of ongoing professional development for the aged care workforce, including The education program was developed in response to discussions with the Victorian Government to help support Victoria’s RACFs during the second wave of COVID-19. A targeted IPC program that specifically focused on application of PPE in the setting of an ever changing COVID-19 infection landscape. The DHHS had already identified the need for this education in RACFs.Prior to implementation of the education program, a pilot of the face-to-face education session was held with staff at a RACF in Melbourne, Victoria. Eight RACF staff attended: a mix of the facility care manager (a registered nurse), registered nurses (RNs), enrolled nurses (ENs) and PCAs. The aim of the pilot was to ensure the content of the education session was appropriate, engaging and useful for RACF staff, as well ensuring timing and sequencing was appropriate for the setting and intended audience. Participants most highly valued the opportunity to practice donning and doffing PPE. They also highly valued the ability to discuss IPC and PPE, and the challenges faced in the aged care sector, raising potential and actual challenges in relation to the prevention of COVID-19, with discussion related to what they would do if they had a confirmed case. The feedback provided from the pilot session was very positive. The participants valued the opportunity to identify the difficulties and develop possible solutions if a positive case of COVID-19 was detected in their RACF. Following the pilot, the face-to-face education session was refined and reduced to 90 min, with an additional 30 min allocated to the train-the-trainer session.The program operated under an education research design to ensure program quality and demonstrate outcomes. Data were collected concurrently with program delivery to inform iterative changes required to program delivery. We used a concurrent triangulation mixed methods design , and empThe COVID-19 pandemic has impacted the conduct of education provision, research and evidence synthesis of pandemic-related research projects that were launched at high speed in large numbers . Risk maTo facilitate smooth implementation of the education program, the following framework guided the approach: (1) provide a rapid response; (2) be safe; (3) be preventive; (4) deliver in person; (5) be inclusive; (6) be flexible; (7) create resilient RACFs; and (8) provide successful and sustainable outcomes .A team of nurse academics from Monash University, with expertise and leadership in IPC, aged care, education evaluation research, clinical training development, and delivery, operations and logistics, worked together to rapidly co-design and implement this large-scale education program designed specifically for RACF staff across Victoria. The importance and urgency of rapid training of RACF staff in IPC and the evidence-based PPE use became heightened with the second wave of COVID-19 pandemic in Australia , 17. PriThe program was offered to all Victorian RACFs via an opt-in model and was promoted by the DHHS via newsletters and direct correspondence. A dedicated Monash University websiteA team of educators worked in pairs to co-deliver each education session to a maximum of 20 participants per session. Within 3 months of commencement 309 face-to-face education sessions and train the trainer sessions were provided across 226 individual RACFs, including 159 RACFs located in Melbourne metropolitan area and 67 in regional Victoria. In total, 377 face-to-face education sessions were provided across 277 individual RACFs, including 170 RACFs located in Melbourne Metropolitan area and 107 in Regional Victoria. As part of this program, 4,219 RACF staff, including 1,207 Facility Champions completed the education .To evaluate the education program, participants were asked to complete a feedback survey 1–5 weeks after completion of their education session. Questions asked related to the usefulness of the session, changes made to IPC practice following the session and general feedback. Overwhelmingly, the feedback was positive with RACF workers reporting more understanding of IPC practices and their application within the facility. A snapshot of the participants’ responses to the process evaluation survey is presented in via an individual coded license which gave workers unlimited access to the VRS software program.Monash University commenced a targeted and intensive roll out of the VRS component in November 2020 to all RACF workers that had attended the face-to-face education session. Access to the VRS was A longer than anticipated time for testing and updating the VRS to align with emerging IPC knowledge and practice related to COVID-19 precautions meant there were some delays with the roll out of the VR component to RACF’s. In an effort to overcome these delays, the education team worked closely with RACF IPC leads and management in the roll out phase to facilitate license provision and VRS access for individual RACF’s. A staggered approach to RACF access allowed for follow up phone and video calls and face-to-face VR support sessions with RACF IPC leads and management. To incentivize RACF workers to engage with the VRS a $500 gift voucher was offered to the top four facilities with the most VR license activations.what went right, what went wrong, and how it could be improved . Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.SD, JM, HR, PR, and DG secured the grant. SD and VT conducted the literature search and drafted the manuscript with support and guidance from JM, HR, PR, and DG. All the authors critically reviewed and contributed to the individual parts of the manuscript, approved the final version, and agreed to be accountable for the content of this work.This project was funded by the Victorian Government through the Department of Health and Human Services Victoria (HHSD/20/350937).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


ImportanceOver-application and interviewing are believed to be widespread in residency recruitment. These may have increased during the 2021 virtual recruitment season. The increase does not correspond to an increase in available residency positions and likely results in more interviews with low probabilities of yielding matches. Prior work demonstrates that such marginal interviews are identifiable ‑ from key explanatory factors like same-state for interviewee and program ‑ in sufficient volume to allow programs to substantially decrease interviews.ObjectiveTo evaluate the importance of same-state relationships in primary care and to determine the extent of over-interviewing in the 2021 virtual recruitment season.DesignThe National Resident Matching Program and Thalamus merged match (outcomes) and interview (explanatory variables) data from primary care specialties . Data were analyzed by logistic regression, trained on the 2017-2020 seasons, and projected on the 2021 season for testing.SettingThe setting was the 2017-2021 main residency matches.ParticipantsThis comprised 4,442 interviewees applying to 167 residency programs in primary care.InterventionThis included the transition to virtual recruitment from in-person recruitment in the 2021 residency recruitment season.MeasurementsA total of 20,415 interviews and 20,791 preferred programs with program and interviewee characteristics and match outcomes were included.ResultsSame-state geographic relations predicted match probability in primary care residency interviews better than medical school/residency affiliation, with 86.0% of interviewees matching consistently with their preferences for the same state. Same-state was more effective than medical school affiliations with programs in predicting matching. Eliminating interviews with less than a 5% probability of matching (upper 95% prediction limit) removed 31.5% of interviews.Conclusions and relevanceThe large number of low-match probability interviews demonstrates over-interviewing in primary care. We suggest that programs eliminate interview offers to applications falling below their chosen match probability threshold. Residency recruitment is a historically challenging marketplace that has been further complicated by the coronavirus disease 2019 (COVID-19) pandemic. Recent adjustments faced by both applicants and programs include a pivot to virtual interviewing, decreased availability of “away rotations,” and conversion of the US Medical Licensing Examination (USMLE) Step 1 score to pass/fail. Application costs vary by specialty but may exceed $5,000 per applicant . Primaryhttps://thalamusgme.com) collaborated to combine interview with match data and examine the key variables driving the conversion of an interview to a match. Thalamus is a cloud-based graduate medical education interview scheduling software and management platform. Authors EL, JR, and SK are shareholders. This study extends previous work in anesthesiology [The National Resident Matching Program (NRMP) and Thalamus , “preferred program” (the first ranked program on an interviewee’s rank order list), and rank order list length . Accreditation Council for Graduate Medical Education identifiers were used for programs and Association of American Medical Colleges identifiers were used for interviewees. Data on interviews, as well as program and interviewee characteristics, were procured from Thalamus’ interview management platform and its application screening and review software, Cortex. All applicants and programs in the Thalamus database were successfully linked to programs and interviewees in the NRMP Match data. Note that an “applicant” to the match by definition attended at least one interview; therefore, the population of match applicants or “interviewees” does not include those who applied to residencies but received no interviews.Data analysesAs in our prior work -10, dataOur analyses addressed two objectives: first  - replication of the finding from our prior work -11, thatThe same-state analysis required complete interviewee state, program state, and match/preference data, available from n1=30,211 interviewees and 1,479 programs.Because of differences among residency markets in the three primary care specialties, we included a fixed effect for specialty. We found that the number of interviews per program divided by the number of matched residents varied considerably in 2021 among specialties, with markedly more interviews per matched position per specialty than in anesthesiology . Because each interviewee can match at only one program, there will be substantially more over-interviewing in primary care than in anesthesiology.For our second analysis, we computed program characteristics, such as average USMLE Step 1 and Step 2CK board scores for program residents (matched interviewees). We removed programs with fewer than five matched residents over five years (at most one per year) with complete characteristic data, to avoid mischaracterizing a program based on an inadequate sample. After we joined and limited the data, our sample comprised n2=20,415 interviews from 4,442 interviewees and 167 programs with all data available and meeting the above criteria. The data were split chronologically into train (2017-2020)/test (2021) sets  outcome by interview characteristics, fit to pre-virtual interviewing data. The model simultaneously estimates the contributions of variables, including programs’ and interviewees’ characteristics, to the probability of each interview resulting in a match. We provide descriptive statistics for regression variables in Supplemental Sections C and D. Predictions from the logistic regression are probabilities of an interview resulting in a match. Using corresponding standard errors in the logit scale, we computed an upper prediction limit for each probability (a measurably conservative estimate of match probability).To make them useful to a training program, the resulting probabilities must be categorized according to some strategy for a program to reduce interviews (i.e. should an interview be taken when there is a <5% chance of matching or less than 1%? And how conservative should we be in our estimates considering the variance between interviews within a program?). Additionally, a program should use the upper prediction interval limit, rather than the estimate of the probability, to account for variation in the prediction. Each program must therefore choose its match probability threshold, beneath which interviewees are not considered, as well as an upper prediction interval limit. We offer several reasonable thresholds  with varying upper prediction interval limits  to provide examples of how to determine a threshold that makes sense for the individual program’s characteristics and risk tolerance .Ethics statementInterviewee and residency program data were gathered from five sources . Those data were merged and de-identified. The University of Iowa Institutional Review Board  determined that the study of this de-identified data does not meet the regulatory definition of human subjects research.From 2021 interview and match data, internal medicine, family medicine, and pediatric residency programs interviewed an average of 32, 41, and 32 applicants per matched position, respectively, despite only 7.5%, 4.3%, and 1.8% of positions, respectively, going unfilled in the 2020 Main Residency Match prior to the Supplemental Offer and Acceptance Program (SOAP). The 2020 Main Residency Match represented the largest NRMP match in history up to that year for each specialty and saw an overall position fill rate of 95.2%.Our primary, inferential analysis of same-state matching showed an overall same-state concordance of 85.9% with 9,714 (32.2%) interviewees both preferring and matching in a residency program within the same state as their medical school and 16,223 (53.7%) interviewees both preferring and matching into a program in a different state. Among those who matched in a different state, 1,915 (6.3%) interviewees preferred same-state (ranked a program in the same state first), but matched out-of-state. A total of 2,359 (7.8%) interviewees ranked programs first preferred out of state but matched in the same state. These results indicate that 83.5% of interviewees who wanted to match in the same state did so. Conversely, Alpha Omega Alpha  membership is a sought-after applicant characteristic, but only 35.2% of interviews involved either programs less than half of whose residents were, and interviewees who were not, AOA members, or programs more than half of whose members were, and interviewees who were, AOA members. Its match-predicting accuracy was only 35.2% /AOA status (average) concordance; see Supplemental Section D). Same medical school program affiliation had match-predicting accuracy comparable to same-state, at 90.4%, but fewer students (14.5%) preferred an affiliated program. Specifically, 3,491 (11.6%) of interviewees preferred and matched an affiliated program (See Note *). Consequently, same-state predicted 178.3% more matches than program-medical school affiliation.We also tested the relative efficacy of geographic relationships related to same-state, specifically neighbors , neighbors’ neighbors , and neighbors’ neighbors’ neighbors . Figure Reductions in interviewsOur second, predictive analysis showed that across primary care specialties in 2021, US Senior MD and DO interviewees submitted rank order lists totaling 122,146 cumulative ranks, with the qualifying study sample accounting for 13,170 interviews. Each rank represents an interview. Table In an ancillary analysis, we found that smaller programs tended to interview many more interviewees per position than larger programs Figure . The smaNote*Discordances were 2,033 (6.7%) interviewees not preferring an affiliated program and matching at the program, and 882 (2.9%) preferring an affiliated program and not matching at the program. An affiliated program connotes that the medical school and the residency share an institutional affiliation.Together, our primary  and secondary (predictive) findings demonstrate substantial over-interviewing in the primary care residency market during the 2021 cycle with virtual interviewing. Though there have been calls to reduce applications and interviews , few evihttps://www.aamc.org/data-reports/interactive-data/eras-statistics-data). For primary care specialties, including internal medicine, applicants use Electronic Residency Application Service (ERAS), which charges fees per application that increase with the number of applications . Eliminating 31.5% of applications (22 applications/applicant on average) at the $26 threshold would remove approximately $14 million in application fees annually. While it is not possible to estimate the extent to which a decrease in interviews would contribute to a decrease in applications (as some applicants may still apply to more programs despite less chances of receiving an interview), a meaningful portion of these 14 million dollars would be saved. Beyond application costs, conducting interviews is an expensive, time-consuming process for programs, for whom reductions would also result in tens of millions of dollars in savings; further, it is a tremendously anxiety-provoking period for applicants [Reducing application volume has substantial financial implications. Our model predicts that 31.5% of interviews in primary care specialties could be eliminated, with only minimal changes to final match lists. As an example, in 2020, 25,885 applicants to internal medicine submitted an average of 71 applications per person, totaling 1,837,835 applications  of interviewees who wanted to match same-state did so. Our data showed that primary care programs interview many more applicants per position than specialty programs . The pothttps://www.nrmp.org/wp-content/uploads/2021/08/MRM-Results_and-Data_2021.pdf). This work should also be extended to assess variation in outcomes among underrepresented in medicine groups and to determine other predictors of matching for applicants and programs. Recent research has highlighted the importance of the UME to GME transition in diversifying the physician workforce [This work has two main limitations: it does not include international medical graduates (IMGs) and does not incorporate data to identify underrepresented groups. To assess same-state geographic effects, this study, by design, had to exclude IMGs who live outside the US, despite the fact that they account for ~ 32.5% of filled positions in 2021 . Small programs are compelled to interview many more candidates per position than large programs to reduce the risks of not filling. Therefore, their probabilities of matching each interviewee are substantially lower than in large programs. Smaller programs also tend to be in less urban geographies with smaller populations. We found that interview reductions driven by the implementation of this algorithm would follow program size in a roughly linear relationship Figure . Thus, wThe transition from medical school to residency is a major source of anxiety for students. We identified large numbers of interviews with a very low likelihood of yielding matches in primary care during the 2021 cycle with virtual interviewing. Further, interviewing is only measured after programs have already filtered applications, implying additional over-application. This is the first study to quantify the number of interviews of marginal utility, implying that tens of thousands of interviews, hundreds of thousands of applications, and tens to hundreds of millions of dollars spent on residency application and recruitment are wasted. The residency application and interview markets require broad reform, which should be anchored in the best available evidence. These reforms will affect where physicians train and practice and will therefore have lasting impacts on American healthcare.