An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> Earned value project management is a well-known management system that integrates cost, schedule and technical performance. It allows the calculation of cost and schedule variances and performance indices and forecasts of project cost and schedule duration. The earned value method provides early indications of project performance to highlight the need for eventual corrective action. Earned value management was originally developed for cost management and has not widely been used for forecasting project duration. However, recent research trends show an increase of interest to use performance indicators for predicting total project duration. In this paper, we give an overview of the state-of-the-art knowledge for this new research trend to bring clarity in the often confusing terminology. The purpose of this paper is three-fold. First, we compare the classic earned value performance indicators SV & SPI with the newly developed earned schedule performance indicators SV(t) & SPI(t). Next, we present a generic schedule forecasting formula applicable in different project situations and compare the three methods from literature to forecast total project duration. Finally, we illustrate the use of each method on a simple one activity example project and on real-life project data. <s> BIB001 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> In this paper, we extensively review and evaluate earned value (EV)-based methods to forecast the total project duration. EV systems have been set up to deal with the complex task of controlling and adjusting the baseline project schedule during execution, taking into account project scope, timed delivery and total project budget. Although EV systems have been proven to provide reliable estimates for the follow-up of cost performance within our project assumptions, they often fail to predict the total duration of the project. We present an extensive simulation study where we carefully control the level of uncertainty in the project, the influence of the project network structure on the accuracy of the forecasts and the time horizon where the EV-based measures provide accurate and reliable results. We assume a project setting where project activities and precedence relations are known in advance and do not consider fundamentally unforeseeable events and/or unknown interactions among various actions that might cause entirely unexpected effects in different project parts. This is the first study that investigates the potential of a recently developed method, the earned schedule method, which improves the connection between EV metrics and the project duration forecasts. <s> BIB002 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> Earned value systems have been setup to deal with the complex task of controlling and adjusting the baseline project schedule during execution, taking into account project scope, timed delivery and total project budget. Although earned value systems have been proven to provide reliable estimates for the follow-up of cost performance within certain project assumptions, it often fails to predict the total duration of the project. In the current paper, a brief overview and summary is given of a simulation study that investigates the potential of three earned value based methods to forecast the final project duration. The study assumes a project setting where project activities and precedence relations are known in advance and do not consider fundamentally unforeseeable events and/or unknown interactions among various actions that might cause entirely unexpected effects in different project parts. This is the first paper in a sequence of papers that summarizes the results of the large simulation study initiated by the study of Vanhoucke and Vandevoorde (2007). Each paper will discuss and highlight another aspect of the simulation study. <s> BIB003 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> This paper presents new simulation results on the forecast accuracy of earned value based metrics to predict a project’s final duration. This is the second paper in a series of papers based on the simulation study initially proposed by Vanhoucke and Vandevoorde (2007b). In a previous manuscript published in the Measurable News (Vanhoucke and Vandevoorde, 2007a), it has been shown that the earned schedule method outperforms, on average, the more traditional earned value based methods to predict the final duration of a project, both for early and late projects. In the current manuscript, the simulation study is extended to new simulation scenarios that measure the influence of inaccuracies in the planned duration estimates for critical and non-critical activities on the accuracy of forecasting methods. <s> BIB004 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> In this third paper about the simulation study to measure the accuracy of three earned value based forecasting techniques to predict the final project duration, we elaborate on the influence of the degree of project criticality on the quality of project duration predictions. Based on simulation results of Vanhoucke and Vandevoorde (2007b) and the network paper of Vanhoucke et al. (2008), we present an indicator to measure the structure of a project network, defined by the distribution of activities and the amount of precedence relations and show that this measure clearly influences the accuracy of duration forecasting. Rather than drawing results on a small set of real-life projects, we present simulation results for a large and diverse set of different project networks to be able to draw general results that hold for a wide variety of projects in practice. <s> BIB005 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> Abstract Earned value management (EVM) has provided methods for predicting the final cost for projects. In large part, these methods have not been improved upon since their beginnings and, with one exception, remain unsubstantiated as to accuracy. At the present time, EVM application guidance does not support prediction of final duration for the schedule component of projects. The objective of this research is to improve the capability of project managers for making informed decisions by providing a reliable forecasting method of the final cost and duration. The method put forth and its evaluation make use of a well established project management method, a recent technique for analyzing schedule performance, and the mathematics of statistics to achieve its purpose – EVM, earned schedule (ES) and statistical prediction and testing methods. The calculation method proposed was studied using data from 12 projects. The results for both final cost and duration are shown to be sufficiently reliable for general application of the forecasting method. The use of the method is encouraged; it may be applied irrespective of the type of work or cost and duration magnitude of the project. <s> BIB006 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> In this paper, I briefly describe the various topics of a new book that I have written as a summary of a large research study performed during the past 5 years. The research described in the book deals with the project performance and control phase of the project life cycle, and the corresponding feedback loop from control to planning and scheduling to take corrective actions when necessary (this is known as project tracking or project monitoring). More precisely, the focus is on a reactive scheduling early warning system by means of Earned Value Management (EVM) and Schedule Risk Analysis (SRA). For an overview of EVM, see e.g. Anbari (2003) or Fleming and Koppelman (2005). For an introduction to SRA, see Hulett (1996). Although EVM has been set up to follow up both time and cost, the majority of the research has been focused on the cost aspect. In the book, I focus on the time dimension which has received relatively less attention in the last decennia. The aim of the research is to measure the project performance sensitivity and the forecast accuracy (both in terms of time and cost, but with a focus on the time dimension) of the existing and newly developed metrics based on the principles of EVM and SRA. The research question boils down to the determination of when and in which cases SRA and EVM could lead to improved project tracking and corrective actions decision making. <s> BIB007 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> Meant to complement rather than compete with the existing books on the subject, this book deals with the project performance and control phases of the project life cycle to present a detailed investigation of the project’s time performance measurement methods and risk analysis techniques in order to evaluate existing and newly developed methods in terms of their abilities to improve the corrective actions decision-making process during project tracking. As readers apply what is learned from the book, EVM practices will become even more effective in project management and cost engineering. Individual chapters look at simulation studies in forecast accuracy, schedule adherence, time sensitivity, activity sensitivity and using top-down or bottom-up project tracking. The book also offers an actual real-life case study, a short tutorial on the EVM use in ProTrack, and conclusions on the relative effectiveness for each technique presented. <s> BIB008 </s> An Overview of Recent Research Results and Future Research Avenues Using Simulation Studies in Project Management <s> Study 2: Forecasting Accuracy. <s> Dynamic scheduling refers to the integration of three important phases in the life cycle of a project: baseline scheduling, schedule risk analysis and project control. In this paper, the efficiency of controlling a project is measured and evaluated using a Monte-Carlo simulation study on fictitious and empirical project data. In the study, the construction of a project baseline schedule acts as a point-of-reference for the schedule risk analysis and project control phases. The sensitivity information obtained by the schedule risk analyses (SRA) and the earned value management (EVM) information obtained during project control serve as early warning control parameters that trigger corrective actions to bring projects back on track in case of problems. The focus in this paper lies on the time performance of a project, and not on the prediction and controlling of the project costs. ::: ::: The contribution of this paper is twofold. First, this paper summarizes the main conclusions of various experiments performed in a large simulation study on the efficiency of project control techniques and the ability to trigger corrective actions in case of project problems. The main purpose of these simulation experiments is to understand why EVM and/or SRA work so well in some projects and fail so miserably in others. This study has been awarded by the International Project Management Association in 2008 on the IPMA world congress in Rome (Italy). Secondly, the paper compares the results obtained on fictitious project data with additional tests performed on a set of real-life data from 8 Belgian companies from various sectors. <s> BIB009
In this section, a simulation study to measure the accuracy of two earned value management (EVM) methods and one earned schedule (ES) method to forecast the final duration of a project in progress is discussed, based on the work presented in Vanhoucke and Vandevoorde BIB002 . This study is a follow-up simulation study of the comparison made by Vandevoorde and Vanhoucke BIB001 where three forecasting methods have been discussed and validated on a small sample of empirical project data. Results of this simulation study have also been reported in follow-up papers published by Vanhoucke and Vandevoorde BIB003 , Vanhoucke  , Vanhoucke and Vandevoorde BIB004 BIB005 , and Vanhoucke BIB007 and in the book by Vanhoucke BIB008 and have been validated using empirical project data from 8 Belgian companies from various sectors BIB009 . Earned value management is a methodology used to measure and communicate the real physical progress of a project and to integrate the three critical elements of project management (scope, time, and cost management). It takes into account the work completed, the time taken, and the costs incurred to complete the project and it helps to evaluate and control project risks by measuring project progress in monetary terms. The basic principles and the use in practice have been comprehensively described in many sources  . EVM relies on the schedule performance index (SPI) to measure the performance of the project duration during progress. Although EVM has been set up to follow up both time and cost, the majority of the research has been focused on the cost aspect (see e.g., the paper written by Fleming and Koppelman  who discuss EVM from a price tag point of view). In 2003, an alternative method, known as earned schedule has been proposed by Lipke  which relies on similar principles of EVM not only but also measures the time performance of projects in progress by an alternative schedule performance index SPI( ) that better measures the real-time performance of projects in progress. The three forecasting methods to forecast the final project duration are known as the planned value method (PVM)  , the earned duration method (EDM)   , and the earned schedule method (ESM)  . A prediction for the final project duration along the progress of the progress using one of these three methods is known as the estimated duration at completion, abbreviated by EAC( ). Each of the three methods can be used in three alternative ways, expressing the assumption about future expected project performance, resulting in 3 * 3 = 9 EAC( ) methods. Unique to this simulation study is the use of the activity distribution functions to simulate activity variation, as discussed in Step 2 of the project control simulation algorithm of Section 2. The simple and easy to use triangular distribution is used to simulate activity duration variation, but its parameters are set in such a way that nine predefined simulation scenarios could be tested. These 9 simulation scenarios are defined based on two parameters. The first is the variation in activity durations that can be defined on the critical and or noncritical activities. A second parameter is the controlled performance measured along the simulation runs measured by the schedule performance index SPI( ) at periodic time intervals for each simulation run. The use of these two parameters results in 9 simulation scenarios that can be classified as follows. True Scenarios. Five of the nine scenarios report an average project duration performance (ahead of schedule, on time, or delay) measured by the periodic SPI( ), and finally result in a real project duration that corresponds to the measured performance. These scenarios are called true scenarios since the measures performance metric SPI( ) measures the true outcome of the project. Misleading Scenarios. Two of the nine scenarios are somewhat misleading since they measure a project ahead of schedule or a project delay, while the project finishes exactly on time. False Scenarios. Two of the nine scenarios are simply wrong since the performance measurement of SPI( ) reports the complete opposite than the final outcome. When a project is reported to be ahead of schedule, it finishes late, while a warning for project delays turned out to result in a project finishing earlier than expected. The reason why different simulation settings are used for critical versus noncritical activities lies at the heart of EVM and is based on the comments made by Jacob  . This author argues that the use of EVM and ES metrics on the project level is dangerous and might lead to wrong conclusions. The reason is that variation in noncritical activities has no real effect on the project duration but is nevertheless measured by the SPI and SPI( ) metrics on the project level and hence might give a false warning signal to the project manager. Consequently, the authors suggest to use the SPI and SPI( ) metrics on the activity level to avoid these errors, and certainly not at higher levels of the work breakdown structure (WBS). This concern has also been raised by other authors and has led to a discussion summarised in papers such as Book   , Jacob  , and Lipke  . Although it is recognised that, at higher WBS levels, effects (delays) of nonperforming activities can be neutralised by well performing activities (ahead of schedule), which might result in masking potential problems, in the simulation study of this section, the advice of these authors has not been followed. Instead, in contradiction to the recommendations of Jacob  , the SPI and SPI( ) indicators are nevertheless measured on the project level, realising that it might lead to wrong conclusions. Therefore, the aim of the study is to test what the impact of this error is on the forecasting accuracy when the performance measures are used at the highest WBS level (i.e., the project level). By splitting the scenarios between critical and noncritical activities, the simulation study can be used to test this known error and its impact on the forecasting accuracy. The reason why these recommendations are ignored is that it is believed that the only approach that can be taken by practitioners is indeed to measure performance on the project level. These measures are used up as early warning signals to detect problems and/or opportunities in an easy and efficient way at high levels in the WBS rather than a simple replacement of the critical path based scheduling tools. This early warning signal, if analysed properly, defines the need to eventually drill down into lower WBS levels. In conjunction with the project schedule, it allows to take corrective actions on activities that are in trouble (especially those tasks that are on the critical path). A similar observation has been made by Lipke et al. BIB006 who also note that detailed schedule analysis is a burdensome activity and, if performed, often can have disrupting effects on the project team. EVM offers calculation methods yielding reliable results on higher WBS levels, which greatly simplify final duration and completion date forecasting. The scope of the study and the used measurement points and resulting output metrics can be summarised as follows: (i) Scope of the study is to compare and validate three EVM/ES techniques (PVM, EDM, and ESM) for forecasting the project duration. (ii) Measurement points are periodic performance metrics (SPI and SPI( )) and the resulting 9 forecasting methods (EAC( )). (iii) Output metrics are mean absolute percentage error (MAPE) and mean percentage error (MPE) to measure the accuracy of the three forecasting methods. Table 1 presents partial results of the forecasting accuracy for the three methods (PVM, EDM, and ESM) along the completion stage of the project and for different project networks. The completion stage is measured as the percentage completed EV/BAC with EV the earned value and BAC the budget at completion. Early, middle, and late stages are defined as 0%, 30%, 30%, 70%, and 70%, 100% percentage completed, respectively. The project network structure is shown by the serial/parallel degree of a project and is measured by the SP indicator discussed in Section 3.1. The column with label "P" represents networks with most activities in parallel while the column with label "S" consists of project networks with mainly serial activities. The "S/P" column is a mix of both and contains both parallel and serial activities. The forecast accuracy is given in the body of the table. Since it is measured by the MAPE, lower numbers denote a higher forecast accuracy. The table clearly shows that all three EVM/ES forecasting methods are more reliable as the number of serial activities increases. More serial projects have more critical activities, and hence, potential errors of project performance on high WBS levels are unlikely to happen since each delay on individual (critical) activities has a real effect on the project duration. Moreover, the ESM outperforms the PVM and EDM along all stages of completion to predict the duration of a project. The table also indicates that the accuracy of all EVM performance measures improves towards the final stages. However, the PVM shows a low accuracy at the final stages, due to the unreliable SPI trend. Indeed, it is known that the SPI goes to one, even for projects ending late, leading to biased results, which is not the case for the SPI( ) metric BIB008  .