To what extent does conformal prediction improve the total gain?
Datasets
We experimented with two publicly available real-life event logs from the same domain, i.e., the banking industry, called  BPIC2017https://doi.org/10.4121/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b and BPIC2012https://data.4tu.nl/articles/dataset/BPI_Challenge_2012/12689204/1. These logs represent the execution of a loan origination process. Tab. REF  shows the key characteristics of these logs. We use them because both are big enough, w.r.t. the number of loan applications. Also, have a specific definition for desired, e.g., when customers accept the offer and sign the contract, and undesired outcomes, e.g., when the bank cancels the application or the customer refuses the offer. Furthermore, they have a clear notion of interventions that may reduce the probability of undesired outcomes, e.g., making a second offer to customers.
<TABLE>The logs contain diverse case and event attributes. We use them in our experiments in addition to other extracted attributes, e.g., the number of sent offers, monthly loan interest, and temporal features, to enrich the logs. Then, we define outcomes according to each case's last activity and determine the intervention according to the Creat_Offer activity for cases labeled with undesired outcomes, as shown in Tab. REF . After that, we extract length prefixes at most the 90th percentile to avoid too long cases. Finally, we apply an aggregate encoding method that attains as much information from the original log and outperforms other encoding techniques according to previous work [1]}—the resulting fixed-size feature vector used as input to train the ML algorithms.

Experimental Setup
The experimental setup to construct an intervention policy with the conformal prediction requires splitting the log temporally into three categories training (\(60\%\) ), calibration (\(20\%\) ), and testing (\(20\%\) ). The training set is used to train predictive and causal models. In contrast, the calibration set is used to build the prediction set, and the testing set evaluates the intervention policy at runtime.
We use a Gradient Boosting Decision Tree (GBDT) algorithm, i.e., Catboost [2]}, to train the predictive model that estimates the probability of cases likely to end undesirably, i.e., \(\mathbb {P}(uout)\) . Also, we use the Orthogonal Random Forest (ORF) algorithm implemented in the EconMlhttps://github.com/microsoft/EconML to train a causal model to estimate the \(CATE\) . We use these methods here, since previous work has shown that both GBDT [1]} and ORF [4]} methods achieve good results w.r.t accuracy in predicting undesired outcomes and estimating the intervention effect.
During runtime, ongoing cases are first filtered into candidates where the \(\mathbb {P}(uout)>0.5\) , \(CATE>0\) , and the \(C(X_{test})=\lbrace uout\rbrace \) . These estimates help decide which cases are more likely to end undesirably with confidence and can be persuaded via the intervention. Then we set the \(C_{uout} =20\) , relatively high, to \(C_{in}=1\)  to estimate the expected gain when allocating resources to candidates.
We use different evaluation measures to evaluate the proposed approach. To evaluate the ICP methods, we use the area under the roc curve (AUC) in addition to the F-score. These metrics are reliable, especially when data is imbalanced, and not biased against one of the outcomes. Additionally, how many cases belong to the \(C(X_{test})\)  when it retains only the undesired outcome? Because our goal is to target cases that will end undesirably with confidence.
For the intervention policy under limited resources, we use the total gain, i.e., the sum of gains obtained from executing the intervention per available resource. Together with the accuracy of allocating resources (accuracy/resource), i.e., the number of cases we allocate resources to when needed, i.e., actually undesired cases, divided by the total number of cases we allocate resources to.

Results
We show the results of the proposed approach by first exploring the effect of the user-defined significance level (\(\alpha \) ) on the prediction set REF . Then we analyze how the total gain improves when resources are finite according to the intervention policy that considers conformal prediction REF .
<FIGURE>In Fig. REF . we show for each non-conformity scorning method how (\(\alpha \) ) affects retaining undesired outcomes only in the prediction set to discuss  REF . We see clearly that for the naive and outcome-balanced methods, the number of cases that belong to the prediction set when retaining only undesired outcomes is maximized when \(\alpha =0.4\)  and \(0.2\)  for BPIC12 and BPIC17, respectively. In contrast, in the adaptive method, \(\alpha =0.9\)  for both logs. This happens because of the way we construct the prediction set for each method. For the naive and outcome-balanced methods, the closer to 0, the higher the \(\hat{q}\) . Accordingly, they are less conservative in adding a particular outcome to the prediction set, which is the opposite w.r.t the adaptive method. Additionally, we observe that these significance levels achieve the highest F-score and \(AUC\)  among all other significance levels, as shown in the supplementary materialhttps://zenodo.org/record/7380386. Hence, we use these levels to evaluate how conformal methods improve the total gain and accuracy/resource.
<FIGURE>To discuss REF , we first compare pure predictive methods that target cases when the \(\mathbb {P}(uout)>0.5\)  [5]}, with and without the \(TotalUncertainty<0.75\)  [4]}, against predictive plus ICP (or conformal) when \(C(X_{test})=\lbrace uout\rbrace \) , see Fig. REF . Here, we consider the gain equal to the benefits we earn from applying the intervention, i.e., CATE. Then, we compare predictive (\(\mathbb {P}(uout)>0.5\)  and the \(TotalUncertainty<0.75\) ) plus \(CATE\)  (when it is above 0) [7]} with predictive (\(\mathbb {P}(uout)>0.5\) ) plus CATE plus conformal, see Fig. REF .
For the BPIC2012 log, the total gain (on the left-hand side) improves when we combine any conformal method with pure predictive in Fig. REF  and \(CATE\)  in Fig. REF . In particular, when resources are minimal, with a remarkable accuracy/resource compared to non-conformal methods. Also, the adaptive conformal method outperforms other methods w.r.t the total gain, and similar to other methods, w.r.t accuracy/resource. This is because the adaptive method's defined \(\hat{q}\)  is much higher than the naive and outcome-balanced methods; accordingly, more conservative in adding outcomes to the prediction set.
Moreover, when resources are not restricted, which is different from the situation in practice, we find that non-conformal methods achieve good gains with reasonable accuracy per resource as conformal methods. However, the conformal methods are more conservative since they constrain the allocation of resources.
<FIGURE>On the other hand, for the BPIC2017 log, the adaptive conformal method enhances the total gain with high accuracy when resources are limited and relaxed. Conversely, when resources are limited, the naive and outcome-balanced methods achieve almost similar gains as non-conformal methods. However, still, all conformal methods outperform non-conformal methods w.r.t accuracy/resource.
The proposed PrPM approach outperforms baselines w.r.t the total gain and accuracy/resource according to Fig. REF  and Fig. REF . Additionally, the supplementary materialhttps://zenodo.org/record/7380386 results show that our approach outperforms the work in [4]}. Then, building an intervention policy with limited resources using conformal prediction improves the performance of PrPM methods, thus, business processes.

Threats to Validity
We used two real-life event logs from the same domain in our evaluation. Thus, the evaluation lack generalizability. Therefore, the evaluation is preliminary and requires more experiments with other logs from other domains to be followed up. However, the field of business process mining needs more public logs with a clear notion of both outcome and intervention.
We assume that cases likely to end undesirably will be treated once, and the intervention effect is accurate and will directly minimize the probability of undesired outcomes. Still, there is a threat to ecological validity since cases can be treated more than once and with different actions or interventions.
According to the literature, we used one type of algorithm for predictive, causal, and conformal methods. However, more experiments need to be done with other algorithms to verify the validity of the developed intervention policy.

Conclusion and Future Work
We introduced a prescriptive process monitoring approach that designs an intervention policy to learn for which cases we should trigger interventions and when to maximize the gain in a situation where resources are finite. The approach leverage a conformal prediction method combined with a predictive model to determine with confidence which cases are likely to end undesirably by creating a prediction set rather than a probability score for each outcome. Together with a causal model to estimate the impact of an intervention on the case outcome.
At runtime, these estimates are given to a resource allocator that filters and ranks cases according to a set of rules to maximize the gain. Empirical evaluation shows that intervention policies with conformal instead of pure predictions outperform non-conformal methods, mainly when resources are finite.
Our proposal uses only one type of intervention and triggers it only once for cases. In practice, business cases can receive multiple interventions with different types. Thus, a direction for future work is to extend the current approach to tackle the problem of: which interventions can be triggered, for which cases, and when to maximize a business value, particularly when resources are limited. This gap can be modeled via a restless multi-armed bandit. Where cases are supposed to transition from one state to another based on the intervention.
Reproducibility. The implementation and source code of the approach, detailed with instructions to replicate the evaluation, can be found at: https://github.com/mshoush/conformal-prescriptive-monitoring.
0.5
